2020-11-05T11:28:31.413 INFO:root:teuthology version: 1.0.0-41b569f 2020-11-05T11:28:31.414 INFO:teuthology.results:subset: '929/9999' 2020-11-05T11:28:31.415 INFO:teuthology.results:seed: '2477' 2020-11-05T16:41:50.231 INFO:root:teuthology version: 1.0.0-41b569f 2020-11-05T16:41:50.266 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-06T02:46:14.514 INFO:teuthology.results:Tests finished! gathering results... 2020-11-06T02:47:13.319 INFO:scrape:Found 446 jobs 2020-11-06T02:47:53.541 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:53.542 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.544 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:53.545 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.545 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:53.678 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:53.679 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.681 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:53.682 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.683 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:53.793 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:53.794 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.796 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:53.797 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.797 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:53.912 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:53.913 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.915 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:53.916 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:53.916 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:54.028 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:54.029 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.031 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:54.032 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.033 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:54.146 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:54.147 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.148 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:54.149 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.150 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:54.263 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:54.264 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.266 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:54.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.268 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:54.378 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:54.380 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.381 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:54.382 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.383 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:54.489 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:54.490 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.492 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:54.493 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.493 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:54.601 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:54.602 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.604 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:54.605 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.605 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:47:54.714 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6954-ge6820d41 (e6820d4151f7302ab23305157765dd0b4fc4e835) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f194e25117b] , 2: (DeleteOp::_begin()+0x4c2) [0x556444323cf2] 2020-11-06T02:47:54.715 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:46:58.494 INFO:teuthology.orchestra.run.smithi165.stdout:} , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:46:58.494+0000 7fc166dae700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.717 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-05T15:42:46.238878+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=46, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-05T15:42:44.226392+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=44, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) , 2020-11-05T15:47:01.180 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2020-11-05T15:42:35.183731+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=42, tm_sec=35, tm_wday=3, tm_yday=310, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=5, tm_hour=15, tm_min=46, tm_sec=56, tm_wday=3, tm_yday=310, tm_isdst=0) 2020-11-06T02:47:54.718 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:23.290+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 , 2020-11-05T15:49:23.455 INFO:teuthology.orchestra.run.smithi165.stdout: , 2020-11-05T15:49:23.454+0000 7f8bc3fff700 -1 WARNING: dec 0 - 1 for counter 94006 would wrap. Setting to 0 2020-11-06T02:47:54.719 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-05T15:49:52.289 INFO:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.JFqwo2JkMf , 2020-11-05T15:49:52.296 INFO:tasks.ceph:Cleaning ceph cluster... , 2020-11-05T15:49:52.296 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-06T02:48:20.711 INFO:scrape:Found 14 distinct failure reasons 2020-11-06T02:48:20.713 INFO:scrape:Failure: "2020-11-05T14:30:00.000125+0000 mon.b (mon.0) 443 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 pool(s) do not have an application enabled" in cluster log 2020-11-06T02:48:20.713 INFO:scrape:128 jobs 2020-11-06T02:48:20.715 INFO:scrape:suites intersection: [] 2020-11-06T02:48:20.715 INFO:scrape:suites union: ['1-install/nautilus', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/9-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'd-thrash/none', 'debug/mgr', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/dashboard/{clusters/{2-node-mgr}', 'rados/monthrash/{ceph', 'rados/multimon/{clusters/6', 'rados/multimon/{clusters/9', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton/{all/peer', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rados/verify/{centos_latest', 'rbd-import-export', 'rbd_api}', 'readwrite', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/dashboard}', 'tasks/mon_recovery', 'tasks/mon_recovery}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'tasks/rados_cls_all', 'tasks/repair_test}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/force-sync-many', 'thrashers/many', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashers/sync', 'thrashers/sync-many', 'thrashosds-health', 'thrashosds-health}', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-agent-big}', 'workloads/cache-agent-small}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps-balanced}', 'workloads/cache-snaps}', 'workloads/cache}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-radosbench}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-many-deletes}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-create-delete}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_5925}', 'workloads/rados_api_tests}', 'workloads/rados_mon_osdmap_prune}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_promote_tests}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/set-chunks-read}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}'] 2020-11-06T02:48:20.716 INFO:scrape: 2020-11-06T02:48:20.716 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-11-06T02:48:20.716 INFO:scrape:7 jobs: ['5592208', '5592392', '5592516', '5592449', '5592272', '5592556', '5592320'] 2020-11-06T02:48:20.717 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-11-06T02:48:20.717 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-11-06T02:48:20.717 INFO:scrape: 2020-11-06T02:48:20.718 INFO:scrape:Failure: SSH connection to smithi082 was lost: 'sudo yum -y install ceph-radosgw' 2020-11-06T02:48:20.718 INFO:scrape:2 jobs: ['5592444', '5592285'] 2020-11-06T02:48:20.718 INFO:scrape:suites intersection: [] 2020-11-06T02:48:20.719 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'openstack', 'openstack}', 'rados', 'rados/perf/{ceph', 'rados/verify/{centos_latest', 'settings/optimized', 'tasks/mon_recovery', 'thrashosds-health}', 'ubuntu_latest', 'validater/valgrind}', 'workloads/radosbench_4M_seq_read}'] 2020-11-06T02:48:20.719 INFO:scrape: 2020-11-06T02:48:20.719 INFO:scrape:Failure: Command failed on smithi050 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' 2020-11-06T02:48:20.720 INFO:scrape:2 jobs: ['5592502', '5592265'] 2020-11-06T02:48:20.720 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/balancer', 'supported-random-distro$/{rhel_8}}'] 2020-11-06T02:48:20.720 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/balancer', 'supported-random-distro$/{rhel_8}}'] 2020-11-06T02:48:20.720 INFO:scrape: 2020-11-06T02:48:20.721 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/mon/caps.sh 2020-11-06T02:48:20.721 INFO:scrape:1 jobs: ['5592342'] 2020-11-06T02:48:20.721 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-hybrid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/force-sync-many', 'workloads/rados_mon_workunits}'] 2020-11-06T02:48:20.722 INFO:scrape: 2020-11-06T02:48:20.722 INFO:scrape:Failure: "2020-11-05T12:40:00.000129+0000 mon.a (mon.0) 1570 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 503/3207 objects degraded (15.684%), 34 pgs degraded, 59 pgs undersized" in cluster log 2020-11-06T02:48:20.723 INFO:scrape:8 jobs: ['5592243', '5592177', '5592576', '5592170', '5592203', '5592337', '5592186', '5592216'] 2020-11-06T02:48:20.723 INFO:scrape:suites intersection: ['rados'] 2020-11-06T02:48:20.723 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/singleton/{all/ec-lost-unfound', 'rados/singleton/{all/lost-unfound', 'rados/singleton/{all/lost-unfound-delete', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'thrashers/careful', 'thrashers/default', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-small-objects}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}'] 2020-11-06T02:48:20.723 INFO:scrape: 2020-11-06T02:48:20.724 INFO:scrape:Failure: "2020-11-05T12:50:00.000156+0000 mon.a (mon.0) 3656 : cluster [ERR] Health detail: HEALTH_ERR 1 auth entities have invalid capabilities; 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds" in cluster log 2020-11-06T02:48:20.724 INFO:scrape:3 jobs: ['5592249', '5592600', '5592369'] 2020-11-06T02:48:20.724 INFO:scrape:suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool'] 2020-11-06T02:48:20.725 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] 2020-11-06T02:48:20.725 INFO:scrape: 2020-11-06T02:48:20.725 INFO:scrape:Failure: {'Failure object was': {'smithi203.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/boost-random-1.66.0-6.el8.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} 2020-11-06T02:48:20.726 INFO:scrape:3 jobs: ['5592311', '5592485', '5592446'] 2020-11-06T02:48:20.726 INFO:scrape:suites intersection: ['rados', 'supported-random-distro$/{centos_8}'] 2020-11-06T02:48:20.727 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/9-mons', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/monthrash/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/one', 'thrashosds-health', 'workloads/cache-agent-small}', 'workloads/pool-create-delete}', 'workloads/radosbench-high-concurrency}'] 2020-11-06T02:48:20.727 INFO:scrape: 2020-11-06T02:48:20.728 INFO:scrape:Dead: b'2020-11-06T02:44:38.009 DEBUG:teuthology.exit:Finished running handlers' 2020-11-06T02:48:20.728 INFO:scrape:1 jobs: ['5592561'] 2020-11-06T02:48:20.728 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/valgrind}'] 2020-11-06T02:48:20.728 INFO:scrape: 2020-11-06T02:48:20.729 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi065 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e6820d4151f7302ab23305157765dd0b4fc4e835 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2020-11-06T02:48:20.729 INFO:scrape:1 jobs: ['5592329'] 2020-11-06T02:48:20.729 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/lockdep}'] 2020-11-06T02:48:20.730 INFO:scrape: 2020-11-06T02:48:20.730 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-11-06T02:48:20.730 INFO:scrape:1 jobs: ['5592535'] 2020-11-06T02:48:20.731 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] 2020-11-06T02:48:20.731 INFO:scrape: 2020-11-06T02:48:20.731 INFO:scrape:Failure: "2020-11-05T14:40:00.000114+0000 mon.b (mon.0) 1720 : cluster [ERR] Health detail: HEALTH_ERR nodeep-scrub flag(s) set; full ratio(s) out of order; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull" in cluster log 2020-11-06T02:48:20.732 INFO:scrape:1 jobs: ['5592453'] 2020-11-06T02:48:20.732 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}'] 2020-11-06T02:48:20.732 INFO:scrape: 2020-11-06T02:48:20.732 INFO:scrape:Failure: "2020-11-05T15:40:00.000073+0000 mon.b (mon.0) 1376 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg peering; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull" in cluster log 2020-11-06T02:48:20.733 INFO:scrape:1 jobs: ['5592523'] 2020-11-06T02:48:20.733 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/radosbench}'] 2020-11-06T02:48:20.733 INFO:scrape: 2020-11-06T02:48:20.734 INFO:scrape:Failure: "2020-11-05T15:50:00.000183+0000 mon.a (mon.0) 1124 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; 1 pool(s) do not have an application enabled" in cluster log 2020-11-06T02:48:20.734 INFO:scrape:1 jobs: ['5592530'] 2020-11-06T02:48:20.734 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/redirect}'] 2020-11-06T02:48:20.735 INFO:scrape: 2020-11-06T02:48:20.755 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/ failed: 155 dead: 5 running: 0 waiting: 0 queued: 0 passed: 286 Fail ================================================================= [5592157] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:32:29 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592157/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592157/ "2020-11-05T11:50:00.000190+0000 mon.a (mon.0) 405 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592160] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:25:38 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592160/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592160/ "2020-11-05T12:00:00.000109+0000 mon.b (mon.0) 2886 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log [5592164] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:26:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592164/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592164/ "2020-11-05T11:50:00.000167+0000 mon.b (mon.0) 255 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592167] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:28:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592167/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592167/ "2020-11-05T12:00:00.000085+0000 mon.a (mon.0) 856 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592169] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/many workloads/pool-create-delete} ----------------------------------------------------------------- time: 00:17:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592169/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592169/ "2020-11-05T11:51:23.994086+0000 mon.f (mon.0) 125 : cluster [WRN] Health detail: HEALTH_WARN 2/9 mons down, quorum f,g,b,h,c,i,e" in cluster log [5592170] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592170/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592170/ "2020-11-05T12:00:00.000189+0000 mon.b (mon.0) 1719 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 556/4312 objects degraded (12.894%), 12 pgs degraded" in cluster log [5592172] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:16:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592172/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592172/ "2020-11-05T12:00:00.000098+0000 mon.a (mon.0) 461 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592177] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:51:26 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592177/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592177/ "2020-11-05T12:00:00.000124+0000 mon.a (mon.0) 301 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 17745/261992 objects degraded (6.773%), 11 pgs degraded, 11 pgs undersized" in cluster log [5592179] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:16:49 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592179/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592179/ "2020-11-05T12:00:00.000137+0000 mon.b (mon.0) 779 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592186] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:36:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592186/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592186/ "2020-11-05T12:10:00.000122+0000 mon.a (mon.0) 1032 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 2 osds down; Reduced data availability: 2 pgs inactive; Degraded data redundancy: 987/2106 objects degraded (46.866%), 10 pgs degraded, 7 pgs undersized; 2 slow ops, oldest one blocked for 46 sec, osd.1 has slow ops" in cluster log [5592190] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:23:38 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592190/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592190/ "2020-11-05T12:10:00.000138+0000 mon.b (mon.0) 1288 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592193] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:06:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592193/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592193/ "2020-11-05T12:30:00.004250+0000 mon.b (mon.0) 1714 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; Slow OSD heartbeats on back (longest 4039.285ms); Slow OSD heartbeats on front (longest 4306.923ms); 1 pool(s) do not have an application enabled" in cluster log [5592198] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:04:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592198/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592198/ "2020-11-05T12:20:00.000123+0000 mon.b (mon.0) 882 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592201] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:23:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592201/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592201/ "2020-11-05T12:20:00.000153+0000 mon.b (mon.0) 1456 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log [5592203] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:52:51 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592203/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592203/ "2020-11-05T12:20:00.000119+0000 mon.a (mon.0) 304 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 18437/193882 objects degraded (9.509%), 8 pgs degraded, 8 pgs undersized" in cluster log [5592205] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:12:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592205/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592205/ "2020-11-05T12:13:16.396991+0000 mon.c (mon.0) 14 : cluster [WRN] Health detail: HEALTH_WARN 1/9 mons down, quorum c,b,f,d,e,i,g,h" in cluster log [5592207] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:27:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592207/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592207/ "2020-11-05T12:30:00.000179+0000 mon.b (mon.0) 1832 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull" in cluster log [5592208] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592208/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592208/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9d704ee6c0a24eff9b985533169d00a1 reached maximum tries (180) after waiting for 180 seconds [5592211] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:30:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592211/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592211/ "2020-11-05T12:20:19.338179+0000 mon.b (mon.0) 52 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log [5592212] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} ----------------------------------------------------------------- time: 00:06:15 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592212/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592212/ "2020-11-05T12:13:13.360013+0000 mon.b (mon.1) 69 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log [5592213] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:34:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592213/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592213/ "2020-11-05T12:30:00.000123+0000 mon.b (mon.0) 1439 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; Degraded data redundancy: 232/2548 objects degraded (9.105%), 8 pgs degraded, 8 pgs undersized" in cluster log [5592216] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:58:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592216/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592216/ "2020-11-05T12:30:00.000112+0000 mon.a (mon.0) 366 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 24534/90378 objects degraded (27.146%), 8 pgs degraded, 8 pgs undersized" in cluster log [5592217] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:38:07 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592217/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592217/ "2020-11-05T12:30:00.000103+0000 mon.a (mon.0) 1222 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592220] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 02:22:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592220/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592220/ "2020-11-05T12:20:00.000112+0000 mon.b (mon.0) 472 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 174/265401 objects degraded (0.066%), 1 pg degraded, 1 pg undersized" in cluster log [5592222] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:31:16 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592222/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592222/ "2020-11-05T12:30:00.000102+0000 mon.a (mon.0) 204 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 7775/133922 objects degraded (5.806%), 21 pgs degraded" in cluster log [5592223] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:57:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592223/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592223/ "2020-11-05T13:00:00.000154+0000 mon.b (mon.0) 6102 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592227] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:17:38 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592227/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592227/ "2020-11-05T12:30:00.000122+0000 mon.b (mon.0) 585 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled" in cluster log [5592233] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:14:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592233/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592233/ "2020-11-05T12:30:00.000163+0000 mon.a (mon.0) 1225 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; 1 pool(s) do not have an application enabled" in cluster log [5592235] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:28:57 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592235/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592235/ "2020-11-05T12:50:00.000208+0000 mon.c (mon.0) 2779 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592240] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:19:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592240/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592240/ "2020-11-05T12:30:00.000191+0000 mon.b (mon.0) 690 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592241] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:22:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592241/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592241/ "2020-11-05T12:40:00.000116+0000 mon.b (mon.0) 2543 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull" in cluster log [5592243] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:19:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592243/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592243/ "2020-11-05T12:40:00.000129+0000 mon.a (mon.0) 1570 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 503/3207 objects degraded (15.684%), 34 pgs degraded, 59 pgs undersized" in cluster log [5592246] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:04:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592246/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592246/ "2020-11-05T13:00:00.000177+0000 mon.b (mon.0) 3334 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 166/682 objects degraded (24.340%), 24 pgs degraded" in cluster log [5592249] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:28:07 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592249/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592249/ "2020-11-05T12:50:00.000156+0000 mon.a (mon.0) 3656 : cluster [ERR] Health detail: HEALTH_ERR 1 auth entities have invalid capabilities; 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds" in cluster log [5592252] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 05:06:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592252/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592252/ "2020-11-05T14:25:40.822958+0000 mon.c (mon.0) 7 : cluster [WRN] Health detail: HEALTH_WARN 4 osds down; 1 host (4 osds) down" in cluster log [5592258] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:22:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592258/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592258/ "2020-11-05T12:50:00.000179+0000 mon.a (mon.0) 1685 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Reduced data availability: 16 pgs inactive, 17 pgs peering; Degraded data redundancy: 20/168 objects degraded (11.905%), 5 pgs degraded" in cluster log [5592259] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:19:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592259/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592259/ "2020-11-05T12:50:00.000157+0000 mon.a (mon.0) 1987 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; Degraded data redundancy: 6/2937 objects degraded (0.204%), 1 pg degraded" in cluster log [5592260] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:21:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592260/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592260/ "2020-11-05T12:46:11.497655+0000 mon.f (mon.0) 110 : cluster [WRN] Health detail: HEALTH_WARN 1/9 mons down, quorum f,a,g,b,h,c,d,e" in cluster log [5592261] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} ----------------------------------------------------------------- time: 00:29:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592261/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592261/ "2020-11-05T12:50:00.000132+0000 mon.b (mon.0) 697 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592262] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:30:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592262/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592262/ "2020-11-05T13:00:00.000104+0000 mon.a (mon.0) 1665 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Reduced data availability: 1 pg inactive, 1 pg peering" in cluster log [5592264] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:33:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592264/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592264/ "2020-11-05T13:00:00.000080+0000 mon.a (mon.0) 1916 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull" in cluster log [5592265] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:12:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592265/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592265/ sentry: https://sentry.ceph.com/ceph/sepia/?query=7a67b8447fbc451f8f5f495e550834e2 Command failed on smithi007 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' [5592272] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:19:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592272/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592272/ sentry: https://sentry.ceph.com/ceph/sepia/?query=a6d4a18e228747c1974116467537c99b reached maximum tries (180) after waiting for 180 seconds [5592278] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:16:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592278/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592278/ "2020-11-05T13:00:00.000124+0000 mon.b (mon.0) 632 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 189/1608 objects degraded (11.754%), 3 pgs degraded" in cluster log [5592283] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:18:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592283/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592283/ "2020-11-05T13:00:00.000113+0000 mon.b (mon.0) 1372 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592285] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:07:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592285/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592285/ sentry: https://sentry.ceph.com/ceph/sepia/?query=1b40830640674e2fb8098547016d1e7f Command failed on smithi141 with status 1: 'sudo yum -y install ceph- radosgw-debuginfo' [5592288] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:22:15 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592288/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592288/ "2020-11-05T13:00:00.000189+0000 mon.b (mon.0) 804 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull" in cluster log [5592289] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:25:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592289/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592289/ "2020-11-05T13:00:00.000154+0000 mon.a (mon.0) 530 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592297] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:28:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592297/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592297/ "2020-11-05T13:10:00.000130+0000 mon.a (mon.0) 767 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592299] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:05:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592299/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592299/ "2020-11-05T13:30:00.000119+0000 mon.a (mon.0) 3478 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592301] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:45:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592301/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592301/ "2020-11-05T13:07:29.355939+0000 mon.b (mon.1) 82 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log [5592307] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:32:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592307/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592307/ "2020-11-05T13:10:00.000100+0000 mon.b (mon.0) 342 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592309] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:32:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592309/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592309/ "2020-11-05T13:20:00.000196+0000 mon.b (mon.0) 2599 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled" in cluster log [5592315] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 02:43:13 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592315/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592315/ "2020-11-05T13:30:00.000183+0000 mon.a (mon.0) 2191 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 1027/126636 objects degraded (0.811%), 3 pgs degraded" in cluster log [5592318] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:10:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592318/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592318/ "2020-11-05T13:10:00.000129+0000 mon.a (mon.0) 353 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled" in cluster log [5592320] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592320/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592320/ sentry: https://sentry.ceph.com/ceph/sepia/?query=212976c85bf843b09665a039f85dfccd reached maximum tries (180) after waiting for 180 seconds [5592324] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:00:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592324/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592324/ "2020-11-05T13:30:00.000101+0000 mon.b (mon.0) 2795 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592328] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:25:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592328/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592328/ "2020-11-05T13:20:00.000188+0000 mon.c (mon.2) 1164 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 166/642 objects degraded (25.857%), 7 pgs degraded" in cluster log [5592329] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592329/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592329/ sentry: https://sentry.ceph.com/ceph/sepia/?query=08efa15aa0c14244945cd9511967bd19 Command failed (workunit test rados/test.sh) on smithi065 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e6820d4151f7302ab23305157765dd0b4fc4e835 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5592333] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:28:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592333/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592333/ "2020-11-05T13:20:00.000099+0000 mon.a (mon.0) 1154 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592334] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:22:13 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592334/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592334/ "2020-11-05T13:30:00.000188+0000 mon.b (mon.0) 1278 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; 1 pool(s) do not have an application enabled" in cluster log [5592336] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:09:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592336/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592336/ "2020-11-05T13:14:50.236290+0000 mon.b (mon.1) 23 : cluster [WRN] Health detail: HEALTH_WARN 1/6 mons down, quorum b,c,d,e,f" in cluster log [5592337] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:24:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592337/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592337/ "2020-11-05T13:20:00.000119+0000 mon.a (mon.0) 797 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull; Degraded data redundancy: 89/4359 objects degraded (2.042%), 2 pgs degraded" in cluster log [5592341] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:24:03 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592341/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592341/ "2020-11-05T13:30:00.000114+0000 mon.b (mon.0) 1355 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592342] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 03:18:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592342/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592342/ sentry: https://sentry.ceph.com/ceph/sepia/?query=8f8b88489ccb4a0fb2aa7efa3670bb0b Command failed (workunit test mon/caps.sh) on smithi133 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e6820d4151f7302ab23305157765dd0b4fc4e835 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh' [5592343] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-radosbench} ----------------------------------------------------------------- time: 02:05:37 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592343/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592343/ "2020-11-05T13:20:00.000111+0000 mon.a (mon.0) 317 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592349] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:05:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592349/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592349/ "2020-11-05T13:30:00.000113+0000 mon.b (mon.0) 570 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592355] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:04:58 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592355/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592355/ "2020-11-05T13:20:00.000155+0000 mon.a (mon.0) 100 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 2 pgs down" in cluster log [5592359] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:24:29 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592359/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592359/ "2020-11-05T13:30:00.000198+0000 mon.a (mon.0) 1680 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 45/507 objects degraded (8.876%), 9 pgs degraded" in cluster log [5592365] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:28:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592365/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592365/ "2020-11-05T13:30:00.000149+0000 mon.a (mon.0) 1427 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592369] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:33:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592369/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592369/ "2020-11-05T13:50:00.000163+0000 mon.a (mon.0) 3712 : cluster [ERR] Health detail: HEALTH_ERR 1 auth entities have invalid capabilities; 1 filesystem is degraded; 1 filesystem has a failed mds daemon; 2 filesystems are offline; 1 filesystem is online with fewer MDS than max_mds" in cluster log [5592373] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:11:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592373/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592373/ "2020-11-05T13:30:00.000106+0000 mon.a (mon.0) 627 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; Reduced data availability: 3 pgs inactive, 4 pgs peering; Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull" in cluster log [5592383] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:24:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592383/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592383/ "2020-11-05T13:40:00.000101+0000 mon.a (mon.0) 758 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592388] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 02:53:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592388/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592388/ "2020-11-05T13:50:00.003323+0000 mon.a (mon.0) 1396 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592389] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:08:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592389/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592389/ "2020-11-05T14:20:00.000107+0000 mon.b (mon.0) 5696 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592392] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:18:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592392/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592392/ sentry: https://sentry.ceph.com/ceph/sepia/?query=904a6de7b6594a5fa6b5707800a26aaf reached maximum tries (180) after waiting for 180 seconds [5592396] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:30:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592396/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592396/ "2020-11-05T13:40:00.000093+0000 mon.a (mon.0) 369 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592400] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:22:26 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592400/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592400/ "2020-11-05T13:40:00.000170+0000 mon.b (mon.0) 451 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592401] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:23:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592401/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592401/ "2020-11-05T13:40:00.000144+0000 mon.b (mon.0) 651 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592403] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:28:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592403/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592403/ "2020-11-05T13:38:01.373679+0000 mon.b (mon.1) 82 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log [5592404] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} ----------------------------------------------------------------- time: 00:24:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592404/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592404/ "2020-11-05T13:50:00.000186+0000 mon.b (mon.0) 1335 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 266/2895 objects degraded (9.188%), 4 pgs degraded" in cluster log [5592405] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:27:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592405/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592405/ "2020-11-05T14:00:00.000182+0000 mon.b (mon.0) 1872 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 481/2838 objects degraded (16.949%), 3 pgs degraded, 14 pgs undersized" in cluster log [5592408] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:20:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592408/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592408/ "2020-11-05T13:50:00.000138+0000 mon.a (mon.0) 1576 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592413] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/repair_test} ----------------------------------------------------------------- time: 00:19:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592413/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592413/ "2020-11-05T13:50:00.000182+0000 mon.a (mon.0) 233 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592421] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:22:47 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592421/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592421/ "2020-11-05T13:50:00.000202+0000 mon.b (mon.1) 1969 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 46/381 objects degraded (12.073%), 14 pgs degraded; 1 pool(s) do not have an application enabled" in cluster log [5592426] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:51:58 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592426/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592426/ "2020-11-05T14:30:00.000118+0000 mon.a (mon.0) 1904 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log [5592429] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:50:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592429/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592429/ "2020-11-05T14:20:00.000110+0000 mon.a (mon.0) 285 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull" in cluster log [5592432] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:28:48 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592432/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592432/ "2020-11-05T14:30:00.000146+0000 mon.c (mon.0) 1627 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592433] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} ----------------------------------------------------------------- time: 00:10:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592433/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592433/ "2020-11-05T14:22:06.985248+0000 mon.a (mon.1) 17 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [5592437] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:34:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592437/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592437/ "2020-11-05T14:40:00.000121+0000 mon.b (mon.0) 2258 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592438] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:18:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592438/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592438/ "2020-11-05T14:30:00.000125+0000 mon.b (mon.0) 443 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 pool(s) do not have an application enabled" in cluster log [5592447] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read} ----------------------------------------------------------------- time: 00:25:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592447/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592447/ "2020-11-05T14:40:00.000143+0000 mon.b (mon.0) 1168 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592448] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:19:02 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592448/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592448/ "2020-11-05T14:40:00.000169+0000 mon.a (mon.0) 998 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log [5592449] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:57 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592449/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592449/ sentry: https://sentry.ceph.com/ceph/sepia/?query=9760e4988f914eca9a500e43e707291b reached maximum tries (180) after waiting for 180 seconds [5592450] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:21:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592450/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592450/ "2020-11-05T14:30:00.000140+0000 mon.a (mon.0) 357 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592453] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:21:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592453/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592453/ "2020-11-05T14:40:00.000114+0000 mon.b (mon.0) 1720 : cluster [ERR] Health detail: HEALTH_ERR nodeep-scrub flag(s) set; full ratio(s) out of order; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull" in cluster log [5592457] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:20:47 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592457/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592457/ "2020-11-05T14:40:00.000132+0000 mon.b (mon.0) 1545 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592459] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:30:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592459/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592459/ "2020-11-05T14:40:00.000119+0000 mon.b (mon.0) 1594 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592460] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:28:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592460/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592460/ "2020-11-05T14:40:00.000159+0000 mon.a (mon.0) 1752 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592461] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:28:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592461/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592461/ "2020-11-05T14:40:00.000218+0000 mon.a (mon.0) 222 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 5908/130280 objects degraded (4.535%), 15 pgs degraded" in cluster log [5592470] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:29:15 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592470/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592470/ "2020-11-05T14:40:00.000082+0000 mon.b (mon.0) 1439 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592473] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:19:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592473/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592473/ "2020-11-05T14:40:00.000114+0000 mon.a (mon.0) 724 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592475] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 01:01:45 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592475/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592475/ "2020-11-05T14:50:00.222230+0000 mon.a (mon.0) 976 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Slow OSD heartbeats on back (longest 10087.365ms); Slow OSD heartbeats on front (longest 11534.558ms); 8 pool(s) do not have an application enabled; 1 slow ops, oldest one blocked for 212 sec, mon.a has slow ops" in cluster log [5592478] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:38:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592478/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592478/ "2020-11-05T14:40:00.000186+0000 mon.b (mon.0) 724 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592479] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:30:17 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592479/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592479/ "2020-11-05T14:40:00.000163+0000 mon.b (mon.0) 1478 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592482] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:25:36 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592482/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592482/ "2020-11-05T14:40:00.000149+0000 mon.b (mon.0) 747 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592486] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/rados_5925} ----------------------------------------------------------------- time: 00:12:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592486/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592486/ "2020-11-05T14:59:56.286087+0000 mon.a (mon.0) 117 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [5592487] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes} ----------------------------------------------------------------- time: 00:19:02 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592487/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592487/ "2020-11-05T15:00:00.000089+0000 mon.b (mon.0) 307 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592488] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:22:42 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592488/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592488/ "2020-11-05T15:10:00.000167+0000 mon.a (mon.0) 1842 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log [5592493] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 04:58:58 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592493/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592493/ "2020-11-05T17:20:00.000163+0000 mon.b (mon.0) 815 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592498] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:19:42 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592498/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592498/ "2020-11-05T15:10:00.000117+0000 mon.a (mon.0) 1881 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 2 osds down; Reduced data availability: 3 pgs inactive; Degraded data redundancy: 1086/3882 objects degraded (27.975%), 3 pgs degraded, 13 pgs undersized" in cluster log [5592501] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:26:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592501/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592501/ "2020-11-05T15:10:00.000177+0000 mon.a (mon.0) 426 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592502] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:12:42 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592502/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592502/ sentry: https://sentry.ceph.com/ceph/sepia/?query=6524747736124636a23dcbb8f30bfabc Command failed on smithi050 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t' [5592504] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:27:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592504/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592504/ "2020-11-05T15:20:00.000176+0000 mon.b (mon.0) 1192 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592510] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:34:26 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592510/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592510/ "2020-11-05T15:20:00.000126+0000 mon.b (mon.0) 508 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592512] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:03:57 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592512/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592512/ "2020-11-05T15:30:00.000194+0000 mon.a (mon.0) 2017 : cluster [ERR] Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds daemon; 1 filesystem is offline" in cluster log [5592515] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:20:08 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592515/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592515/ "2020-11-05T15:30:00.000172+0000 mon.b (mon.0) 1128 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log [5592516] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:18:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592516/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592516/ sentry: https://sentry.ceph.com/ceph/sepia/?query=19dbc4f466ee4f6b914e96b6ead47572 reached maximum tries (180) after waiting for 180 seconds [5592518] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:23:27 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592518/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592518/ "2020-11-05T15:30:00.000168+0000 mon.a (mon.0) 1150 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull" in cluster log [5592520] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 01:05:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592520/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592520/ "2020-11-05T15:30:00.000118+0000 mon.a (mon.0) 1024 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 295/1962 objects degraded (15.036%), 9 pgs degraded" in cluster log [5592523] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 02:21:57 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592523/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592523/ "2020-11-05T15:40:00.000073+0000 mon.b (mon.0) 1376 : cluster [WRN] Health detail: HEALTH_WARN Reduced data availability: 1 pg peering; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull" in cluster log [5592525] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:27:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592525/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592525/ "2020-11-05T15:40:00.000125+0000 mon.b (mon.0) 1465 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull" in cluster log [5592526] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:25:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592526/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592526/ "2020-11-05T15:40:00.000094+0000 mon.a (mon.0) 859 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592530] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:18:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592530/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592530/ "2020-11-05T15:50:00.000183+0000 mon.a (mon.0) 1124 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; 1 pool(s) do not have an application enabled" in cluster log [5592532] rados/multimon/{clusters/6 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:16:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592532/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592532/ "2020-11-05T15:37:03.216375+0000 mon.b (mon.0) 70 : cluster [WRN] Health detail: HEALTH_WARN 1/6 mons down, quorum b,a,d,f,e" in cluster log [5592533] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:23:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592533/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592533/ "2020-11-05T15:40:38.357106+0000 mon.f (mon.0) 119 : cluster [WRN] Health detail: HEALTH_WARN 3/9 mons down, quorum f,g,h,c,i,e" in cluster log [5592534] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:19:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592534/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592534/ "2020-11-05T15:40:00.000124+0000 mon.b (mon.0) 346 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592535] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:17:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592535/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592535/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5592536] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} ----------------------------------------------------------------- time: 00:11:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592536/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592536/ "2020-11-05T15:40:00.000193+0000 mon.b (mon.0) 238 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled" in cluster log [5592538] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:16:42 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592538/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592538/ "2020-11-05T16:00:00.000121+0000 mon.b (mon.0) 385 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; 1 pool(s) do not have an application enabled" in cluster log [5592539] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard} ----------------------------------------------------------------- time: 00:59:08 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592539/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592539/ "2020-11-05T15:50:00.000133+0000 mon.a (mon.0) 1509 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592541] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:17:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592541/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592541/ "2020-11-05T16:00:00.000124+0000 mon.a (mon.0) 146 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log [5592545] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:22:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592545/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592545/ "2020-11-05T16:00:00.000094+0000 mon.a (mon.0) 1311 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592548] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:22:47 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592548/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592548/ "2020-11-05T16:00:00.000135+0000 mon.b (mon.0) 1494 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log [5592553] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:22:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592553/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592553/ "2020-11-05T16:00:00.000167+0000 mon.b (mon.0) 172 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592556] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592556/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592556/ sentry: https://sentry.ceph.com/ceph/sepia/?query=c9a6edafacec4b0eb92b16d56dd23246 reached maximum tries (180) after waiting for 180 seconds [5592557] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:25:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592557/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592557/ "2020-11-05T16:10:00.000106+0000 mon.a (mon.0) 925 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592559] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:27:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592559/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592559/ "2020-11-05T16:10:00.000120+0000 mon.a (mon.0) 737 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592563] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:00:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592563/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592563/ "2020-11-05T16:30:00.000174+0000 mon.b (mon.0) 3303 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log [5592569] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:22:58 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592569/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592569/ "2020-11-05T16:20:00.000096+0000 mon.a (mon.0) 1302 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592573] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:41:13 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592573/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592573/ "2020-11-05T16:16:38.458415+0000 mon.b (mon.1) 53 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log [5592576] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:23:29 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592576/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592576/ "2020-11-05T16:20:00.000108+0000 mon.a (mon.0) 752 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 221/1632 objects degraded (13.542%), 10 pgs degraded, 10 pgs undersized" in cluster log [5592579] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:31:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592579/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592579/ "2020-11-05T16:30:00.000120+0000 mon.a (mon.0) 759 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592583] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:15:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592583/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592583/ "2020-11-05T16:30:00.000119+0000 mon.b (mon.0) 249 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592586] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:26:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592586/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592586/ "2020-11-05T16:40:00.000093+0000 mon.b (mon.0) 1427 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 414/2040 objects degraded (20.294%), 4 pgs degraded" in cluster log [5592590] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:25:10 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592590/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592590/ "2020-11-05T16:30:00.000169+0000 mon.b (mon.0) 288 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set" in cluster log [5592591] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:24:36 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592591/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592591/ "2020-11-05T16:40:00.000139+0000 mon.b (mon.0) 1597 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592598] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:24:49 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592598/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592598/ "2020-11-05T17:10:00.000101+0000 mon.b (mon.0) 1001 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set" in cluster log [5592600] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:30:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592600/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592600/ "2020-11-05T17:00:00.000130+0000 mon.a (mon.0) 3585 : cluster [ERR] Health detail: HEALTH_ERR 1 auth entities have invalid capabilities; 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds" in cluster log [5592602] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:16:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592602/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592602/ "2020-11-05T16:50:00.000081+0000 mon.a (mon.0) 211 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set" in cluster log [5592606] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:30:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592606/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592606/ "2020-11-05T16:50:00.000183+0000 mon.c (mon.0) 1438 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 81/828 objects degraded (9.783%), 19 pgs degraded" in cluster log [5592607] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/lockdep} ----------------------------------------------------------------- time: 00:10:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592607/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592607/ "2020-11-05T16:47:21.234190+0000 mon.a (mon.1) 17 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log [5592609] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:33:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592609/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592609/ "2020-11-05T16:50:00.000122+0000 mon.b (mon.0) 271 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Reduced data availability: 5 pgs inactive; Degraded data redundancy: 7/80 objects degraded (8.750%), 3 pgs degraded" in cluster log [5592611] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:25:49 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592611/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592611/ "2020-11-05T17:00:00.000141+0000 mon.b (mon.0) 2797 : cluster [WRN] Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; noscrub,nodeep- scrub flag(s) set; 2 pool(s) do not have an application enabled" in cluster log [5592613] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} ----------------------------------------------------------------- time: 01:04:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592613/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592613/ "2020-11-05T17:40:00.000179+0000 mon.b (mon.0) 5218 : cluster [WRN] Health detail: HEALTH_WARN no active mgr" in cluster log Dead ================================================================= [5592311] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:02:04 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592311/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592311/ sentry: https://sentry.ceph.com/ceph/sepia/?query=b376107d270d4fd990457c8e2737c82d {'Failure object was': {'smithi203.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/boost- random-1.66.0-6.el8.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu- kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} [5592444] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} ----------------------------------------------------------------- time: 00:04:16 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592444/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592444/ sentry: https://sentry.ceph.com/ceph/sepia/?query=3bf9d278a12e42e4baca275d9f688e07 SSH connection to smithi082 was lost: 'sudo yum -y install ceph-radosgw' [5592446] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/one workloads/pool-create-delete} ----------------------------------------------------------------- time: 00:04:04 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592446/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592446/ sentry: https://sentry.ceph.com/ceph/sepia/?query=c8fed62770de4b8d965dad6c4a1865a8 {'Failure object was': {'smithi082.front.sepia.ceph.com': {'msg': 'Failed to manage policy for boolean nagios_run_sudo: [Errno 11] Resource temporarily unavailable', 'invocation': {'module_args': {'name': 'nagios_run_sudo', 'state': True, 'persistent': True, 'ignore_selinux_state': False}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'nagios_run_sudo')"} [5592485] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} ----------------------------------------------------------------- time: 00:03:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592485/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592485/ sentry: https://sentry.ceph.com/ceph/sepia/?query=b180298385da4c23b229e8630c511e2a {'Failure object was': {'smithi003.front.sepia.ceph.com': {'msg': 'Failed to download packages: Cannot download Packages/qemu-kvm-block- rbd-2.12.0-88.module_el8.1.0+297+df420408.3.x86_64.rpm: All mirrors were tried', 'results': [], 'invocation': {'module_args': {'name': ['dnf-utils', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'podman', 'podman-docker', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm- core', 'qemu-kvm-block-rbd', 'libacl-devel', 'dbench', 'autoconf', 'gdb', 'iozone'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com _git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_te uthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthwo rker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_master/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '/')"} [5592561] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592561/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592561/ Pass ================================================================= [5592153] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:24:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592153/ [5592154] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mon} time: 00:43:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592154/ [5592155] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:22:37 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592155/ [5592156] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/classic} time: 00:16:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592156/ [5592158] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592158/ [5592159] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:08:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592159/ [5592161] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:23:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592161/ [5592162] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} time: 01:43:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592162/ [5592163] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:04:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592163/ [5592165] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:07:04 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592165/ [5592166] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli} time: 00:16:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592166/ [5592168] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:10:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592168/ [5592171] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:15 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592171/ [5592173] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:10:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592173/ [5592174] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_stress_watch} time: 00:19:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592174/ [5592175] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 00:59:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592175/ [5592176] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:27:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592176/ [5592178] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:24:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592178/ [5592180] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592180/ [5592181] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:11:07 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592181/ [5592182] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:19:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592182/ [5592183] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/osd} time: 03:02:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592183/ [5592184] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:25:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592184/ [5592185] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:21:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592185/ [5592187] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:04:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592187/ [5592188] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:31:57 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592188/ [5592189] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:15:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592189/ [5592191] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/progress} time: 00:20:10 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592191/ [5592192] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592192/ [5592194] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:28:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592194/ [5592195] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:11:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592195/ [5592196] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:10 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592196/ [5592197] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:16 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592197/ [5592199] rados/objectstore/{backends/objectstore supported-random-distro$/{centos_8}} time: 05:05:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592199/ [5592200] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:17:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592200/ [5592202] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:35:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592202/ [5592204] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:15:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592204/ [5592206] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:38:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592206/ [5592209] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:05:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592209/ [5592210] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:06:05 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592210/ [5592214] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:10:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592214/ [5592215] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:06:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592215/ [5592218] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/scrub} time: 01:33:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592218/ [5592219] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:12:45 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592219/ [5592221] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:14:51 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592221/ [5592224] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/redirect} time: 00:14:24 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592224/ [5592225] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:04:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592225/ [5592226] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:27 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592226/ [5592228] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:05 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592228/ [5592229] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:11:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592229/ [5592230] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:20:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592230/ [5592231] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592231/ [5592232] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/prometheus} time: 00:08:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592232/ [5592234] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:09:36 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592234/ [5592236] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:13:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592236/ [5592237] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592237/ [5592238] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:12:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592238/ [5592239] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:08:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592239/ [5592242] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:29:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592242/ [5592244] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:05:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592244/ [5592245] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} time: 00:28:16 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592245/ [5592247] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:06:10 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592247/ [5592248] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:06:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592248/ [5592250] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592250/ [5592251] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/crush} time: 00:08:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592251/ [5592253] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:16:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592253/ [5592254] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm} time: 00:11:36 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592254/ [5592255] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects} time: 00:16:02 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592255/ [5592256] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:16:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592256/ [5592257] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:14:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592257/ [5592263] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:09:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592263/ [5592266] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592266/ [5592267] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:06:49 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592267/ [5592268] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:12:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592268/ [5592269] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:09:45 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592269/ [5592270] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:11:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592270/ [5592271] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:37:38 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592271/ [5592273] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/dashboard} time: 00:58:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592273/ [5592274] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-big} time: 00:16:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592274/ [5592275] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:06:36 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592275/ [5592276] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:32:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592276/ [5592277] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:08:37 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592277/ [5592279] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/workunits} time: 00:11:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592279/ [5592280] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:53:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592280/ [5592281] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:40:16 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592281/ [5592282] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:07 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592282/ [5592284] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592284/ [5592286] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:11:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592286/ [5592287] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} time: 00:18:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592287/ [5592290] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592290/ [5592291] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:10:51 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592291/ [5592292] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:03:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592292/ [5592293] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:16:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592293/ [5592294] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:04:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592294/ [5592295] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:08:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592295/ [5592296] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:19:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592296/ [5592298] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:16:02 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592298/ [5592300] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:09:04 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592300/ [5592302] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:23:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592302/ [5592303] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache} time: 00:08:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592303/ [5592304] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592304/ [5592305] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:07:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592305/ [5592306] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:09:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592306/ [5592308] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:20:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592308/ [5592310] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:14:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592310/ [5592312] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:08:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592312/ [5592313] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:17:16 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592313/ [5592314] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:04:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592314/ [5592316] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:35:51 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592316/ [5592317] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:20:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592317/ [5592319] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:08:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592319/ [5592321] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/crash} time: 00:07:48 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592321/ [5592322] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592322/ [5592323] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:29:08 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592323/ [5592325] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:09:13 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592325/ [5592326] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592326/ [5592327] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:17:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592327/ [5592330] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592330/ [5592331] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:11:47 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592331/ [5592332] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592332/ [5592335] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:29:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592335/ [5592338] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_orch_cli} time: 00:16:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592338/ [5592339] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:12:02 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592339/ [5592340] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592340/ [5592344] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:18:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592344/ [5592345] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/misc} time: 00:28:05 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592345/ [5592346] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:20:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592346/ [5592347] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592347/ [5592348] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592348/ [5592351] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:04:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592351/ [5592353] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592353/ [5592357] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:26:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592357/ [5592361] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:06:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592361/ [5592363] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:24:48 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592363/ [5592367] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:15:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592367/ [5592371] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592371/ [5592375] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:13:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592375/ [5592377] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/failover} time: 00:15:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592377/ [5592379] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592379/ [5592381] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:14:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592381/ [5592384] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:12:48 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592384/ [5592385] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592385/ [5592386] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:19:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592386/ [5592387] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:51 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592387/ [5592390] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592390/ [5592391] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:10:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592391/ [5592393] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:21:13 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592393/ [5592394] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:41:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592394/ [5592395] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:05:43 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592395/ [5592397] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_adoption} time: 00:05:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592397/ [5592398] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:12:04 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592398/ [5592399] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:11:03 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592399/ [5592402] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:29:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592402/ [5592406] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:35:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592406/ [5592407] rados/singleton/{all/radostool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:12:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592407/ [5592409] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:11:04 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592409/ [5592410] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592410/ [5592411] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache} time: 00:09:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592411/ [5592412] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:10:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592412/ [5592414] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:02:47 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592414/ [5592415] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:23:10 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592415/ [5592416] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:16:24 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592416/ [5592417] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:21:15 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592417/ [5592418] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592418/ [5592419] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:20:29 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592419/ [5592420] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:28:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592420/ [5592422] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592422/ [5592423] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592423/ [5592424] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592424/ [5592425] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/insights} time: 00:16:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592425/ [5592427] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:07:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592427/ [5592428] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:12:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592428/ [5592430] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:19:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592430/ [5592431] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:21:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592431/ [5592434] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:12:30 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592434/ [5592435] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:41:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592435/ [5592436] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:19:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592436/ [5592439] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:05:43 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592439/ [5592440] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:15:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592440/ [5592441] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:10:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592441/ [5592442] rados/multimon/{clusters/21 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:07:26 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592442/ [5592443] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:23:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592443/ [5592445] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:13:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592445/ [5592451] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592451/ [5592452] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:12:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592452/ [5592454] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:15:08 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592454/ [5592455] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:09:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592455/ [5592456] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/scrub} time: 01:34:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592456/ [5592458] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:22:18 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592458/ [5592462] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:34:51 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592462/ [5592463] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:43 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592463/ [5592464] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:05:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592464/ [5592465] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:17:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592465/ [5592466] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:25:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592466/ [5592467] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:26:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592467/ [5592468] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:15:43 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592468/ [5592469] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:10:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592469/ [5592471] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:09:08 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592471/ [5592472] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:13:08 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592472/ [5592474] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:15:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592474/ [5592476] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:12:49 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592476/ [5592477] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm_repos} time: 00:05:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592477/ [5592480] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:32:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592480/ [5592481] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:19:23 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592481/ [5592483] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:14:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592483/ [5592484] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:06:43 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592484/ [5592489] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:04:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592489/ [5592490] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:28:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592490/ [5592491] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:17:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592491/ [5592492] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:08:37 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592492/ [5592494] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:20:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592494/ [5592495] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:17:42 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592495/ [5592496] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 01:31:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592496/ [5592497] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:23:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592497/ [5592499] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:23:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592499/ [5592500] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:20:05 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592500/ [5592503] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:09:35 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592503/ [5592505] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592505/ [5592506] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:34:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592506/ [5592507] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592507/ [5592508] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache} time: 00:12:06 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592508/ [5592509] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/progress} time: 00:15:28 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592509/ [5592511] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:11:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592511/ [5592513] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:05:04 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592513/ [5592514] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:46 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592514/ [5592517] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:22:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592517/ [5592519] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_cls_all validater/lockdep} time: 00:13:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592519/ [5592521] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:36:02 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592521/ [5592522] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:17:01 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592522/ [5592524] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/sample_fio} time: 00:10:52 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592524/ [5592527] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:11:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592527/ [5592528] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:19:17 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592528/ [5592529] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:24 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592529/ [5592531] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:16:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592531/ [5592537] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:30:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592537/ [5592540] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592540/ [5592542] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:04:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592542/ [5592543] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:10:45 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592543/ [5592544] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:07:03 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592544/ [5592546] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592546/ [5592547] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:39:56 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592547/ [5592549] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:08:25 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592549/ [5592550] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:09:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592550/ [5592551] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:15:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592551/ [5592552] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mgr} time: 00:09:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592552/ [5592554] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592554/ [5592555] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:05:12 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592555/ [5592558] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption} time: 00:06:34 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592558/ [5592560] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:08 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592560/ [5592562] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:12:10 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592562/ [5592564] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:12:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592564/ [5592565] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:19:14 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592565/ [5592566] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:24:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592566/ [5592567] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:22:27 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592567/ [5592568] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:06:50 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592568/ [5592570] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:11:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592570/ [5592571] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:11:59 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592571/ [5592572] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:20:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592572/ [5592574] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:06:33 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592574/ [5592575] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:25:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592575/ [5592577] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:13:19 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592577/ [5592578] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:20:20 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592578/ [5592580] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:37 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592580/ [5592581] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592581/ [5592582] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:08:21 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592582/ [5592584] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:14:17 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592584/ [5592585] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:22:38 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592585/ [5592587] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:11 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592587/ [5592588] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:30:44 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592588/ [5592589] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:04:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592589/ [5592592] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:34:39 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592592/ [5592593] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/workunits} time: 00:12:01 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592593/ [5592594] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:12:10 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592594/ [5592595] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:06:53 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592595/ [5592596] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:21:41 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592596/ [5592597] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:13:17 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592597/ [5592599] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_stress_watch} time: 00:20:31 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592599/ [5592601] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:13:47 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592601/ [5592603] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:26:54 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592603/ [5592604] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:12:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592604/ [5592605] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:08:22 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592605/ [5592608] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:55 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592608/ [5592610] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 02:05:09 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592610/ [5592612] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:40 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592612/ [5592614] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{rhel_8}} time: 00:26:00 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592614/ [5592615] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:32 info: http://pulpito.ceph.com/kchai-2020-11-05_11:13:13-rados-wip-kefu-testing-2020-11-05-1414-distro-basic-smithi/5592615/