2021-01-10T14:21:24.838 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-10T14:21:24.841 INFO:teuthology.results:subset: '961/9999' 2021-01-10T14:21:24.841 INFO:teuthology.results:seed: '2477' 2021-01-10T18:05:47.328 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-10T18:05:47.364 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-11T04:36:12.664 INFO:teuthology.results:Tests finished! gathering results... 2021-01-11T04:36:59.383 INFO:scrape:Found 446 jobs 2021-01-11T04:37:01.320 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f392dae8aaa] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55ba73f5e306] 2021-01-11T04:37:01.322 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:07:33.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-10T18:07:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: , dumped all 2021-01-11T04:37:01.323 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:08:16.477 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 , 2021-01-10T18:08:16.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com , 2021-01-10T18:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-01-11T04:37:01.323 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:10:24.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com , 2021-01-10T18:10:24.814 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-01-10T18:10:24.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2021-01-11T04:37:01.420 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f392dae8aaa] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55ba73f5e306] 2021-01-11T04:37:01.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:07:33.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-10T18:07:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: , dumped all 2021-01-11T04:37:01.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:08:16.477 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 , 2021-01-10T18:08:16.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com , 2021-01-10T18:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-01-11T04:37:01.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:10:24.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com , 2021-01-10T18:10:24.814 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-01-10T18:10:24.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2021-01-11T04:37:01.523 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f392dae8aaa] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55ba73f5e306] 2021-01-11T04:37:01.524 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:07:33.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-10T18:07:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: , dumped all 2021-01-11T04:37:01.525 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:08:16.477 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 , 2021-01-10T18:08:16.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com , 2021-01-10T18:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-01-11T04:37:01.525 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:10:24.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com , 2021-01-10T18:10:24.814 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-01-10T18:10:24.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2021-01-11T04:37:01.632 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f392dae8aaa] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55ba73f5e306] 2021-01-11T04:37:01.634 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:07:33.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-10T18:07:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: , dumped all 2021-01-11T04:37:01.634 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:08:16.477 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 , 2021-01-10T18:08:16.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com , 2021-01-10T18:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-01-11T04:37:01.635 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:10:24.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com , 2021-01-10T18:10:24.814 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-01-10T18:10:24.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2021-01-11T04:37:01.734 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f392dae8aaa] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55ba73f5e306] 2021-01-11T04:37:01.736 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:07:33.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-10T18:07:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: , dumped all 2021-01-11T04:37:01.737 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:08:16.477 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 , 2021-01-10T18:08:16.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com , 2021-01-10T18:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-01-11T04:37:01.737 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:10:24.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com , 2021-01-10T18:10:24.814 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-01-10T18:10:24.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2021-01-11T04:37:01.845 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f392dae8aaa] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55ba73f5e306] 2021-01-11T04:37:01.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:07:33.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-10T18:07:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: , dumped all 2021-01-11T04:37:01.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:08:16.477 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 , 2021-01-10T18:08:16.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com , 2021-01-10T18:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-01-11T04:37:01.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:10:24.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com , 2021-01-10T18:10:24.814 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-01-10T18:10:24.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2021-01-11T04:37:01.950 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f392dae8aaa] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1306) [0x55ba73f5e306] 2021-01-11T04:37:01.951 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:07:33.797 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-10T18:07:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: , dumped all 2021-01-11T04:37:01.952 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:08:16.477 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 , 2021-01-10T18:08:16.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com , 2021-01-10T18:08:16.585 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-01-11T04:37:01.953 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T18:10:24.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com , 2021-01-10T18:10:24.814 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-01-10T18:10:24.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2021-01-11T04:37:11.828 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faab3df6aaa] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55616b80ffe6] 2021-01-11T04:37:11.830 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.957 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:38.512712+0000", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:11.830 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:40.421421+0000", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.68076948199999998 2021-01-11T04:37:11.831 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:31.002 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:39.741116+0000", 2021-01-11T04:37:11.831 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "throttled", 2021-01-11T04:37:11.832 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:31.050 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:11.833 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.0969000000000004e-05 , 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:11.833 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:55.345211+0000", , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 6.7390000000000002e-06 , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:11.834 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42288184299999998, , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "reached pg", 2021-01-11T04:37:11.834 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:08.296149+0000", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42611334499999998 2021-01-11T04:37:11.835 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:07.887617+0000", 2021-01-11T04:37:11.835 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", 2021-01-11T04:37:11.836 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.484 INFO:teuthology.orchestra.run.smithi154.stdout:} , 2021-01-10T15:54:31.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops , 2021-01-10T15:54:31.540+0000 7fe453023700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34712) UID: 0 2021-01-11T04:37:11.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:31.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:22.414585+0000", 2021-01-11T04:37:11.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:27.706385+0000", 2021-01-11T04:37:11.838 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:29.931673+0000", 2021-01-11T04:37:11.838 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:30.124946+0000", 2021-01-11T04:37:11.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2021-01-11T04:37:11.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:11.840 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6830 3.2 3:5cd4ad39:::smithi04439965-124 low_tier:head [stat] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e47)", , 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:53:03.742354+0000", , 2021-01-10T15:54:31.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.881855142000006, 2021-01-11T04:37:11.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:03.761715+0000", 2021-01-11T04:37:11.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.376657+0000", 2021-01-11T04:37:11.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:11.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7835 4.12 4:4db4c399:::smithi04439965-19:head [read 0~2302526,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e97)", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:21.374658+0000", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.249551077, 2021-01-11T04:37:11.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.489e-06 , 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:11.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2021-01-11T04:37:11.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:24.959496+0000", , 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3161e-05 , 2021-01-10T15:54:31.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:11.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4410", , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.44:0/1539467638", 2021-01-11T04:37:11.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2021-01-11T04:37:11.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:11.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1226999999999998e-05 , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:11.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2021-01-11T04:37:11.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.351231+0000", , 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.66597844500000003 , 2021-01-10T15:54:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:11.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.851114+0000", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.799e-06 2021-01-11T04:37:11.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.409547+0000", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.52158149499999995 2021-01-11T04:37:11.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:32.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991047+0000", 2021-01-11T04:37:11.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991504+0000", 2021-01-11T04:37:11.850 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:11.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:32.575 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:11.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026621799999999998 , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:11.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:11.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2021-01-11T04:37:11.853 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:11.853 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2021-01-11T04:37:11.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:11.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:4965 4.f 4:fb2c833d:::smithi04439965-282:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", , 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:52:39.049386+0000", , 2021-01-10T15:54:33.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.95760128800001, 2021-01-11T04:37:11.855 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5874999999999997e-05 , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:11.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:44.207697+0000", , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1967000000000001e-05 , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:11.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6939 4.f 4:fc3e770b:::smithi04439965-357:head [copy-from ver 75 in=56b] snapc 0=[] ondisk+write+known_if_redirected e48)", 2021-01-11T04:37:11.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:05.606654+0000", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027123799999999998 2021-01-11T04:37:11.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.869699+0000", 2021-01-11T04:37:11.858 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:09.512586+0000", , 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.494401447000001, , 2021-01-10T15:54:33.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.110411859, 2021-01-11T04:37:11.858 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:11.859 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6205000000000002e-05 , 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:11.859 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275366+0000", , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013393500000000001 , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:11.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6983 4.b 4:d781c871:::smithi04439965-386:head [copy-from ver 69 in=56b] snapc 0=[] RETRY=7 ondisk+retry+write+known_if_redirected e82)", 2021-01-11T04:37:11.861 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275769+0000", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3127e-05 2021-01-11T04:37:11.861 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.348376+0000", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.065188815 2021-01-11T04:37:11.862 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.283408+0000", 2021-01-11T04:37:11.862 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:11.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:11.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7220 4.b 4:dc09a8ef:::smithi04439965-412:head [sparse-read 0~2907677] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected e82)", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:07.284227+0000", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.50491452, 2021-01-11T04:37:11.864 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8408000000000001e-05 , 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:11.865 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.293923+0000", , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069715399999999996 , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:11.865 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.018 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7048 4.9 4:91c960fe:::smithi04439965-346:head [read 0~1] snapc 0=[] RETRY=6 ondisk+retry+read+rwordered+known_if_redirected e83)", 2021-01-11T04:37:11.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.990606+0000", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4151999999999999e-05 2021-01-11T04:37:11.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.992209+0000", 2021-01-11T04:37:11.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798724+0000", , 2021-01-10T15:54:34.527 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 9.0710000000000001e-06 2021-01-11T04:37:11.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798639+0000", 2021-01-11T04:37:11.868 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:11.868 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", 2021-01-11T04:37:11.869 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00044863999999999999 , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:11.870 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "client_info": { 2021-01-11T04:37:11.870 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:21.464420+0000", , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.571733563 , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:11.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.699 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.514150+0000", , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.17075385200000001 , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:11.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.739 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343544+0000", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:11.872 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.781 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343773+0000", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00023528000000000001 2021-01-11T04:37:11.873 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.369290+0000", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.8164000000000006e-05 2021-01-11T04:37:11.873 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.505067+0000", , 2021-01-10T15:54:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.6768999999999996e-05 2021-01-11T04:37:11.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:30.203847+0000", , 2021-01-10T15:54:34.874 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.200877156 2021-01-11T04:37:11.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:34.284046+0000", 2021-01-11T04:37:11.876 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:53.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats , 2021-01-10T15:54:53.393 INFO:teuthology.orchestra.run.smithi044.stdout:60129542183 , 2021-01-10T15:54:53.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-01-11T04:37:11.878 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:04.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 , instructed osd(s) 6 to deep-scrub , 2021-01-10T15:55:04.650 INFO:tasks.ceph:Scrubbing osd.7 2021-01-11T04:37:11.879 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu2-testing-2021-01-10-1949/qa/tasks/ceph.py", line 1883, in task , yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 176, in run_tasks 2021-01-11T04:37:11.879 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:50.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.549 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.Qk6NhAVPxy is 947KB 2021-01-11T04:37:11.880 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:57:00.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-01-10T15:57:00.301 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-01-10T15:57:00.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-01-11T04:37:12.024 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faab3df6aaa] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55616b80ffe6] 2021-01-11T04:37:12.025 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.957 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:38.512712+0000", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.025 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:40.421421+0000", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.68076948199999998 2021-01-11T04:37:12.026 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:31.002 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:39.741116+0000", 2021-01-11T04:37:12.027 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "throttled", 2021-01-11T04:37:12.027 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:31.050 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.028 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.0969000000000004e-05 , 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.028 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:55.345211+0000", , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 6.7390000000000002e-06 , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.029 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42288184299999998, , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "reached pg", 2021-01-11T04:37:12.030 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:08.296149+0000", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42611334499999998 2021-01-11T04:37:12.030 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:07.887617+0000", 2021-01-11T04:37:12.031 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", 2021-01-11T04:37:12.031 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.484 INFO:teuthology.orchestra.run.smithi154.stdout:} , 2021-01-10T15:54:31.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops , 2021-01-10T15:54:31.540+0000 7fe453023700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34712) UID: 0 2021-01-11T04:37:12.032 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:31.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:22.414585+0000", 2021-01-11T04:37:12.033 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:27.706385+0000", 2021-01-11T04:37:12.033 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:29.931673+0000", 2021-01-11T04:37:12.034 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:30.124946+0000", 2021-01-11T04:37:12.034 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2021-01-11T04:37:12.035 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.035 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6830 3.2 3:5cd4ad39:::smithi04439965-124 low_tier:head [stat] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e47)", , 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:53:03.742354+0000", , 2021-01-10T15:54:31.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.881855142000006, 2021-01-11T04:37:12.036 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:03.761715+0000", 2021-01-11T04:37:12.036 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.376657+0000", 2021-01-11T04:37:12.037 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.038 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7835 4.12 4:4db4c399:::smithi04439965-19:head [read 0~2302526,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e97)", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:21.374658+0000", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.249551077, 2021-01-11T04:37:12.038 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.489e-06 , 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.039 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2021-01-11T04:37:12.039 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:24.959496+0000", , 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3161e-05 , 2021-01-10T15:54:31.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.040 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4410", , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.44:0/1539467638", 2021-01-11T04:37:12.040 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2021-01-11T04:37:12.041 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.042 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1226999999999998e-05 , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.042 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2021-01-11T04:37:12.043 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.351231+0000", , 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.66597844500000003 , 2021-01-10T15:54:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.043 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.851114+0000", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.799e-06 2021-01-11T04:37:12.044 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.409547+0000", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.52158149499999995 2021-01-11T04:37:12.044 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:32.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991047+0000", 2021-01-11T04:37:12.045 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991504+0000", 2021-01-11T04:37:12.046 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.046 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:32.575 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.047 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026621799999999998 , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.047 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.048 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2021-01-11T04:37:12.048 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.049 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.050 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.050 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:4965 4.f 4:fb2c833d:::smithi04439965-282:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", , 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:52:39.049386+0000", , 2021-01-10T15:54:33.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.95760128800001, 2021-01-11T04:37:12.051 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5874999999999997e-05 , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.052 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:44.207697+0000", , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1967000000000001e-05 , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.052 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6939 4.f 4:fc3e770b:::smithi04439965-357:head [copy-from ver 75 in=56b] snapc 0=[] ondisk+write+known_if_redirected e48)", 2021-01-11T04:37:12.053 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:05.606654+0000", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027123799999999998 2021-01-11T04:37:12.054 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.869699+0000", 2021-01-11T04:37:12.054 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:09.512586+0000", , 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.494401447000001, , 2021-01-10T15:54:33.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.110411859, 2021-01-11T04:37:12.055 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.055 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6205000000000002e-05 , 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.056 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275366+0000", , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013393500000000001 , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.056 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6983 4.b 4:d781c871:::smithi04439965-386:head [copy-from ver 69 in=56b] snapc 0=[] RETRY=7 ondisk+retry+write+known_if_redirected e82)", 2021-01-11T04:37:12.057 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275769+0000", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3127e-05 2021-01-11T04:37:12.057 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.348376+0000", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.065188815 2021-01-11T04:37:12.058 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.283408+0000", 2021-01-11T04:37:12.059 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.059 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.060 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7220 4.b 4:dc09a8ef:::smithi04439965-412:head [sparse-read 0~2907677] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected e82)", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:07.284227+0000", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.50491452, 2021-01-11T04:37:12.060 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8408000000000001e-05 , 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.061 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.293923+0000", , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069715399999999996 , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.061 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.018 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7048 4.9 4:91c960fe:::smithi04439965-346:head [read 0~1] snapc 0=[] RETRY=6 ondisk+retry+read+rwordered+known_if_redirected e83)", 2021-01-11T04:37:12.062 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.990606+0000", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4151999999999999e-05 2021-01-11T04:37:12.062 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.992209+0000", 2021-01-11T04:37:12.063 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798724+0000", , 2021-01-10T15:54:34.527 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 9.0710000000000001e-06 2021-01-11T04:37:12.063 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798639+0000", 2021-01-11T04:37:12.064 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.065 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.065 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00044863999999999999 , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.066 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "client_info": { 2021-01-11T04:37:12.066 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:21.464420+0000", , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.571733563 , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.067 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.699 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.514150+0000", , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.17075385200000001 , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.067 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.739 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343544+0000", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.068 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.781 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343773+0000", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00023528000000000001 2021-01-11T04:37:12.068 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.369290+0000", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.8164000000000006e-05 2021-01-11T04:37:12.069 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.505067+0000", , 2021-01-10T15:54:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.6768999999999996e-05 2021-01-11T04:37:12.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:30.203847+0000", , 2021-01-10T15:54:34.874 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.200877156 2021-01-11T04:37:12.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:34.284046+0000", 2021-01-11T04:37:12.072 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:53.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats , 2021-01-10T15:54:53.393 INFO:teuthology.orchestra.run.smithi044.stdout:60129542183 , 2021-01-10T15:54:53.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-01-11T04:37:12.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:04.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 , instructed osd(s) 6 to deep-scrub , 2021-01-10T15:55:04.650 INFO:tasks.ceph:Scrubbing osd.7 2021-01-11T04:37:12.074 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu2-testing-2021-01-10-1949/qa/tasks/ceph.py", line 1883, in task , yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 176, in run_tasks 2021-01-11T04:37:12.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:50.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.549 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.Qk6NhAVPxy is 947KB 2021-01-11T04:37:12.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:57:00.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-01-10T15:57:00.301 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-01-10T15:57:00.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-01-11T04:37:12.205 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faab3df6aaa] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55616b80ffe6] 2021-01-11T04:37:12.207 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.957 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:38.512712+0000", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.207 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:40.421421+0000", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.68076948199999998 2021-01-11T04:37:12.208 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:31.002 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:39.741116+0000", 2021-01-11T04:37:12.208 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "throttled", 2021-01-11T04:37:12.209 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:31.050 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.209 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.0969000000000004e-05 , 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.210 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:55.345211+0000", , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 6.7390000000000002e-06 , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.211 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42288184299999998, , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "reached pg", 2021-01-11T04:37:12.211 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:08.296149+0000", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42611334499999998 2021-01-11T04:37:12.212 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:07.887617+0000", 2021-01-11T04:37:12.212 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", 2021-01-11T04:37:12.213 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.484 INFO:teuthology.orchestra.run.smithi154.stdout:} , 2021-01-10T15:54:31.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops , 2021-01-10T15:54:31.540+0000 7fe453023700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34712) UID: 0 2021-01-11T04:37:12.214 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:31.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:22.414585+0000", 2021-01-11T04:37:12.214 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:27.706385+0000", 2021-01-11T04:37:12.215 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:29.931673+0000", 2021-01-11T04:37:12.215 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:30.124946+0000", 2021-01-11T04:37:12.216 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2021-01-11T04:37:12.216 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.217 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6830 3.2 3:5cd4ad39:::smithi04439965-124 low_tier:head [stat] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e47)", , 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:53:03.742354+0000", , 2021-01-10T15:54:31.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.881855142000006, 2021-01-11T04:37:12.218 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:03.761715+0000", 2021-01-11T04:37:12.218 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.376657+0000", 2021-01-11T04:37:12.219 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.219 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7835 4.12 4:4db4c399:::smithi04439965-19:head [read 0~2302526,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e97)", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:21.374658+0000", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.249551077, 2021-01-11T04:37:12.220 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.489e-06 , 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.220 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2021-01-11T04:37:12.221 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:24.959496+0000", , 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3161e-05 , 2021-01-10T15:54:31.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.222 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4410", , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.44:0/1539467638", 2021-01-11T04:37:12.222 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2021-01-11T04:37:12.223 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.223 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1226999999999998e-05 , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.224 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2021-01-11T04:37:12.225 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.351231+0000", , 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.66597844500000003 , 2021-01-10T15:54:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.225 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.851114+0000", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.799e-06 2021-01-11T04:37:12.226 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.409547+0000", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.52158149499999995 2021-01-11T04:37:12.226 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:32.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991047+0000", 2021-01-11T04:37:12.227 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991504+0000", 2021-01-11T04:37:12.228 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.228 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:32.575 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.229 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026621799999999998 , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.229 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.230 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2021-01-11T04:37:12.231 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.231 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.232 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.232 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:4965 4.f 4:fb2c833d:::smithi04439965-282:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", , 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:52:39.049386+0000", , 2021-01-10T15:54:33.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.95760128800001, 2021-01-11T04:37:12.233 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5874999999999997e-05 , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.233 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:44.207697+0000", , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1967000000000001e-05 , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.234 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6939 4.f 4:fc3e770b:::smithi04439965-357:head [copy-from ver 75 in=56b] snapc 0=[] ondisk+write+known_if_redirected e48)", 2021-01-11T04:37:12.235 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:05.606654+0000", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027123799999999998 2021-01-11T04:37:12.235 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.869699+0000", 2021-01-11T04:37:12.236 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:09.512586+0000", , 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.494401447000001, , 2021-01-10T15:54:33.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.110411859, 2021-01-11T04:37:12.236 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.237 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6205000000000002e-05 , 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.237 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275366+0000", , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013393500000000001 , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.238 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6983 4.b 4:d781c871:::smithi04439965-386:head [copy-from ver 69 in=56b] snapc 0=[] RETRY=7 ondisk+retry+write+known_if_redirected e82)", 2021-01-11T04:37:12.239 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275769+0000", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3127e-05 2021-01-11T04:37:12.239 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.348376+0000", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.065188815 2021-01-11T04:37:12.240 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.283408+0000", 2021-01-11T04:37:12.240 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.241 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.241 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7220 4.b 4:dc09a8ef:::smithi04439965-412:head [sparse-read 0~2907677] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected e82)", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:07.284227+0000", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.50491452, 2021-01-11T04:37:12.242 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8408000000000001e-05 , 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.243 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.293923+0000", , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069715399999999996 , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.243 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.018 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7048 4.9 4:91c960fe:::smithi04439965-346:head [read 0~1] snapc 0=[] RETRY=6 ondisk+retry+read+rwordered+known_if_redirected e83)", 2021-01-11T04:37:12.244 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.990606+0000", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4151999999999999e-05 2021-01-11T04:37:12.244 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.992209+0000", 2021-01-11T04:37:12.245 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798724+0000", , 2021-01-10T15:54:34.527 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 9.0710000000000001e-06 2021-01-11T04:37:12.245 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798639+0000", 2021-01-11T04:37:12.246 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.247 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.247 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00044863999999999999 , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.248 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "client_info": { 2021-01-11T04:37:12.248 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:21.464420+0000", , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.571733563 , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.249 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.699 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.514150+0000", , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.17075385200000001 , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.249 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.739 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343544+0000", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.250 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.781 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343773+0000", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00023528000000000001 2021-01-11T04:37:12.251 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.369290+0000", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.8164000000000006e-05 2021-01-11T04:37:12.251 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.505067+0000", , 2021-01-10T15:54:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.6768999999999996e-05 2021-01-11T04:37:12.252 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:30.203847+0000", , 2021-01-10T15:54:34.874 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.200877156 2021-01-11T04:37:12.252 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:34.284046+0000", 2021-01-11T04:37:12.254 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:53.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats , 2021-01-10T15:54:53.393 INFO:teuthology.orchestra.run.smithi044.stdout:60129542183 , 2021-01-10T15:54:53.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-01-11T04:37:12.256 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:04.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 , instructed osd(s) 6 to deep-scrub , 2021-01-10T15:55:04.650 INFO:tasks.ceph:Scrubbing osd.7 2021-01-11T04:37:12.257 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu2-testing-2021-01-10-1949/qa/tasks/ceph.py", line 1883, in task , yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 176, in run_tasks 2021-01-11T04:37:12.257 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:50.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.549 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.Qk6NhAVPxy is 947KB 2021-01-11T04:37:12.258 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:57:00.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-01-10T15:57:00.301 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-01-10T15:57:00.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-01-11T04:37:12.391 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faab3df6aaa] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55616b80ffe6] 2021-01-11T04:37:12.392 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.957 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:38.512712+0000", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.393 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:40.421421+0000", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.68076948199999998 2021-01-11T04:37:12.393 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:31.002 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:39.741116+0000", 2021-01-11T04:37:12.394 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "throttled", 2021-01-11T04:37:12.394 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:31.050 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.395 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.0969000000000004e-05 , 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.396 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:55.345211+0000", , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 6.7390000000000002e-06 , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.396 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42288184299999998, , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "reached pg", 2021-01-11T04:37:12.397 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:08.296149+0000", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42611334499999998 2021-01-11T04:37:12.397 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:07.887617+0000", 2021-01-11T04:37:12.398 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", 2021-01-11T04:37:12.398 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.484 INFO:teuthology.orchestra.run.smithi154.stdout:} , 2021-01-10T15:54:31.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops , 2021-01-10T15:54:31.540+0000 7fe453023700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34712) UID: 0 2021-01-11T04:37:12.399 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:31.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:22.414585+0000", 2021-01-11T04:37:12.400 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:27.706385+0000", 2021-01-11T04:37:12.400 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:29.931673+0000", 2021-01-11T04:37:12.401 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:30.124946+0000", 2021-01-11T04:37:12.401 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2021-01-11T04:37:12.402 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.402 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6830 3.2 3:5cd4ad39:::smithi04439965-124 low_tier:head [stat] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e47)", , 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:53:03.742354+0000", , 2021-01-10T15:54:31.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.881855142000006, 2021-01-11T04:37:12.403 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:03.761715+0000", 2021-01-11T04:37:12.404 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.376657+0000", 2021-01-11T04:37:12.404 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.405 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7835 4.12 4:4db4c399:::smithi04439965-19:head [read 0~2302526,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e97)", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:21.374658+0000", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.249551077, 2021-01-11T04:37:12.405 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.489e-06 , 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.406 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2021-01-11T04:37:12.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:24.959496+0000", , 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3161e-05 , 2021-01-10T15:54:31.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4410", , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.44:0/1539467638", 2021-01-11T04:37:12.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2021-01-11T04:37:12.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1226999999999998e-05 , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2021-01-11T04:37:12.410 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.351231+0000", , 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.66597844500000003 , 2021-01-10T15:54:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.851114+0000", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.799e-06 2021-01-11T04:37:12.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.409547+0000", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.52158149499999995 2021-01-11T04:37:12.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:32.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991047+0000", 2021-01-11T04:37:12.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991504+0000", 2021-01-11T04:37:12.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:32.575 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026621799999999998 , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2021-01-11T04:37:12.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:4965 4.f 4:fb2c833d:::smithi04439965-282:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", , 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:52:39.049386+0000", , 2021-01-10T15:54:33.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.95760128800001, 2021-01-11T04:37:12.418 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5874999999999997e-05 , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:44.207697+0000", , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1967000000000001e-05 , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6939 4.f 4:fc3e770b:::smithi04439965-357:head [copy-from ver 75 in=56b] snapc 0=[] ondisk+write+known_if_redirected e48)", 2021-01-11T04:37:12.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:05.606654+0000", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027123799999999998 2021-01-11T04:37:12.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.869699+0000", 2021-01-11T04:37:12.421 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:09.512586+0000", , 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.494401447000001, , 2021-01-10T15:54:33.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.110411859, 2021-01-11T04:37:12.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6205000000000002e-05 , 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275366+0000", , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013393500000000001 , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6983 4.b 4:d781c871:::smithi04439965-386:head [copy-from ver 69 in=56b] snapc 0=[] RETRY=7 ondisk+retry+write+known_if_redirected e82)", 2021-01-11T04:37:12.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275769+0000", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3127e-05 2021-01-11T04:37:12.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.348376+0000", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.065188815 2021-01-11T04:37:12.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.283408+0000", 2021-01-11T04:37:12.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7220 4.b 4:dc09a8ef:::smithi04439965-412:head [sparse-read 0~2907677] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected e82)", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:07.284227+0000", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.50491452, 2021-01-11T04:37:12.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8408000000000001e-05 , 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.293923+0000", , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069715399999999996 , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.018 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7048 4.9 4:91c960fe:::smithi04439965-346:head [read 0~1] snapc 0=[] RETRY=6 ondisk+retry+read+rwordered+known_if_redirected e83)", 2021-01-11T04:37:12.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.990606+0000", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4151999999999999e-05 2021-01-11T04:37:12.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.992209+0000", 2021-01-11T04:37:12.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798724+0000", , 2021-01-10T15:54:34.527 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 9.0710000000000001e-06 2021-01-11T04:37:12.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798639+0000", 2021-01-11T04:37:12.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00044863999999999999 , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "client_info": { 2021-01-11T04:37:12.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:21.464420+0000", , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.571733563 , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.699 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.514150+0000", , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.17075385200000001 , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.739 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343544+0000", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.781 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343773+0000", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00023528000000000001 2021-01-11T04:37:12.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.369290+0000", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.8164000000000006e-05 2021-01-11T04:37:12.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.505067+0000", , 2021-01-10T15:54:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.6768999999999996e-05 2021-01-11T04:37:12.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:30.203847+0000", , 2021-01-10T15:54:34.874 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.200877156 2021-01-11T04:37:12.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:34.284046+0000", 2021-01-11T04:37:12.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:53.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats , 2021-01-10T15:54:53.393 INFO:teuthology.orchestra.run.smithi044.stdout:60129542183 , 2021-01-10T15:54:53.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-01-11T04:37:12.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:04.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 , instructed osd(s) 6 to deep-scrub , 2021-01-10T15:55:04.650 INFO:tasks.ceph:Scrubbing osd.7 2021-01-11T04:37:12.442 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu2-testing-2021-01-10-1949/qa/tasks/ceph.py", line 1883, in task , yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 176, in run_tasks 2021-01-11T04:37:12.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:50.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.549 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.Qk6NhAVPxy is 947KB 2021-01-11T04:37:12.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:57:00.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-01-10T15:57:00.301 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-01-10T15:57:00.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-01-11T04:37:12.568 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7faab3df6aaa] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55616b80ffe6] 2021-01-11T04:37:12.569 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.957 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:38.512712+0000", , 2021-01-10T15:54:30.958 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.569 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:40.421421+0000", , 2021-01-10T15:54:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.68076948199999998 2021-01-11T04:37:12.570 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.001 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:31.002 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:39.741116+0000", 2021-01-11T04:37:12.571 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "throttled", 2021-01-11T04:37:12.571 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:31.049 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:31.050 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.572 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.0969000000000004e-05 , 2021-01-10T15:54:31.072 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.073 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.572 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:52:55.345211+0000", , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 6.7390000000000002e-06 , 2021-01-10T15:54:31.095 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.573 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42288184299999998, , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:31.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "reached pg", 2021-01-11T04:37:12.573 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:08.296149+0000", , 2021-01-10T15:54:31.139 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.42611334499999998 2021-01-11T04:37:12.574 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:31.159 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:07.887617+0000", 2021-01-11T04:37:12.574 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:31.179 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", 2021-01-11T04:37:12.575 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.484 INFO:teuthology.orchestra.run.smithi154.stdout:} , 2021-01-10T15:54:31.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops , 2021-01-10T15:54:31.540+0000 7fe453023700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 34712) UID: 0 2021-01-11T04:37:12.575 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:31.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:22.414585+0000", 2021-01-11T04:37:12.576 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2021-01-10T15:54:31.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:27.706385+0000", 2021-01-11T04:37:12.577 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:29.931673+0000", 2021-01-11T04:37:12.577 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:30.124946+0000", 2021-01-11T04:37:12.578 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2021-01-11T04:37:12.578 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.579 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6830 3.2 3:5cd4ad39:::smithi04439965-124 low_tier:head [stat] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e47)", , 2021-01-10T15:54:31.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:53:03.742354+0000", , 2021-01-10T15:54:31.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.881855142000006, 2021-01-11T04:37:12.579 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", , 2021-01-10T15:54:31.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:03.761715+0000", 2021-01-11T04:37:12.580 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", , 2021-01-10T15:54:31.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.376657+0000", 2021-01-11T04:37:12.580 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:31.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.581 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7835 4.12 4:4db4c399:::smithi04439965-19:head [read 0~2302526,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e97)", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:21.374658+0000", , 2021-01-10T15:54:31.862 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.249551077, 2021-01-11T04:37:12.581 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.489e-06 , 2021-01-10T15:54:31.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:31.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.582 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2021-01-11T04:37:12.583 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:24.959496+0000", , 2021-01-10T15:54:31.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3161e-05 , 2021-01-10T15:54:31.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.583 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4410", , 2021-01-10T15:54:32.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.44:0/1539467638", 2021-01-11T04:37:12.584 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2021-01-11T04:37:12.584 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.585 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1226999999999998e-05 , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.585 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", , 2021-01-10T15:54:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2021-01-11T04:37:12.586 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.351231+0000", , 2021-01-10T15:54:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.66597844500000003 , 2021-01-10T15:54:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.587 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.851114+0000", , 2021-01-10T15:54:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.799e-06 2021-01-11T04:37:12.587 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.409547+0000", , 2021-01-10T15:54:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.52158149499999995 2021-01-11T04:37:12.588 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:32.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991047+0000", 2021-01-11T04:37:12.588 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2021-01-10T15:54:32.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.991504+0000", 2021-01-11T04:37:12.589 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:32.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.590 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:32.574 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:32.575 INFO:teuthology.orchestra.run.smithi154.stdout: } 2021-01-11T04:37:12.590 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:32.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026621799999999998 , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:32.595 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.591 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ , 2021-01-10T15:54:33.014 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2021-01-11T04:37:12.591 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.039 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2021-01-11T04:37:12.592 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.090 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.592 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.593 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.594 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:4965 4.f 4:fb2c833d:::smithi04439965-282:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", , 2021-01-10T15:54:33.161 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:52:39.049386+0000", , 2021-01-10T15:54:33.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.95760128800001, 2021-01-11T04:37:12.594 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5874999999999997e-05 , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.595 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:52:44.207697+0000", , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1967000000000001e-05 , 2021-01-10T15:54:33.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.595 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.251 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6939 4.f 4:fc3e770b:::smithi04439965-357:head [copy-from ver 75 in=56b] snapc 0=[] ondisk+write+known_if_redirected e48)", 2021-01-11T04:37:12.596 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:53:05.606654+0000", , 2021-01-10T15:54:33.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027123799999999998 2021-01-11T04:37:12.596 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:33.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.869699+0000", 2021-01-11T04:37:12.597 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:09.512586+0000", , 2021-01-10T15:54:33.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.494401447000001, , 2021-01-10T15:54:33.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.110411859, 2021-01-11T04:37:12.597 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2021-01-11T04:37:12.598 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6205000000000002e-05 , 2021-01-10T15:54:33.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.599 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275366+0000", , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013393500000000001 , 2021-01-10T15:54:33.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.599 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:6983 4.b 4:d781c871:::smithi04439965-386:head [copy-from ver 69 in=56b] snapc 0=[] RETRY=7 ondisk+retry+write+known_if_redirected e82)", 2021-01-11T04:37:12.600 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.275769+0000", , 2021-01-10T15:54:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3127e-05 2021-01-11T04:37:12.600 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:08.348376+0000", , 2021-01-10T15:54:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.065188815 2021-01-11T04:37:12.601 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2021-01-10T15:54:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.283408+0000", 2021-01-11T04:37:12.601 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: ] , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: } , 2021-01-10T15:54:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.602 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2021-01-11T04:37:12.602 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7220 4.b 4:dc09a8ef:::smithi04439965-412:head [sparse-read 0~2907677] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected e82)", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2021-01-10T15:54:07.284227+0000", , 2021-01-10T15:54:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.50491452, 2021-01-11T04:37:12.603 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8408000000000001e-05 , 2021-01-10T15:54:33.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:33.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2021-01-11T04:37:12.603 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:07.293923+0000", , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069715399999999996 , 2021-01-10T15:54:33.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2021-01-11T04:37:12.604 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2021-01-10T15:54:34.017 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.018 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4410.0:7048 4.9 4:91c960fe:::smithi04439965-346:head [read 0~1] snapc 0=[] RETRY=6 ondisk+retry+read+rwordered+known_if_redirected e83)", 2021-01-11T04:37:12.604 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.990606+0000", , 2021-01-10T15:54:34.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4151999999999999e-05 2021-01-11T04:37:12.605 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", , 2021-01-10T15:54:34.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2021-01-10T15:54:13.992209+0000", 2021-01-11T04:37:12.606 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.526 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798724+0000", , 2021-01-10T15:54:34.527 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 9.0710000000000001e-06 2021-01-11T04:37:12.606 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "header_read", , 2021-01-10T15:54:34.550 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:11.798639+0000", 2021-01-11T04:37:12.607 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: ] , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: } , 2021-01-10T15:54:34.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.607 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.608 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", 2021-01-11T04:37:12.608 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00044863999999999999 , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: }, , 2021-01-10T15:54:34.631 INFO:teuthology.orchestra.run.smithi044.stdout: { 2021-01-11T04:37:12.608 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "type_data": { , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-10T15:54:34.653 INFO:teuthology.orchestra.run.smithi044.stdout: "client_info": { 2021-01-11T04:37:12.609 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:21.464420+0000", , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.571733563 , 2021-01-10T15:54:34.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2021-01-11T04:37:12.609 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.699 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.514150+0000", , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.17075385200000001 , 2021-01-10T15:54:34.700 INFO:teuthology.orchestra.run.smithi044.stdout: } 2021-01-11T04:37:12.610 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.739 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "initiated", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343544+0000", , 2021-01-10T15:54:34.740 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0 2021-01-11T04:37:12.610 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.781 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "queued_for_pg", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.343773+0000", , 2021-01-10T15:54:34.782 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.00023528000000000001 2021-01-11T04:37:12.611 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.369290+0000", , 2021-01-10T15:54:34.815 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.8164000000000006e-05 2021-01-11T04:37:12.611 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "reached_pg", , 2021-01-10T15:54:34.850 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:25.505067+0000", , 2021-01-10T15:54:34.851 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 7.6768999999999996e-05 2021-01-11T04:37:12.612 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "done", , 2021-01-10T15:54:34.873 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:30.203847+0000", , 2021-01-10T15:54:34.874 INFO:teuthology.orchestra.run.smithi044.stdout: "duration": 0.200877156 2021-01-11T04:37:12.613 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: { , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "event": "dispatched", , 2021-01-10T15:54:34.896 INFO:teuthology.orchestra.run.smithi044.stdout: "time": "2021-01-10T15:54:34.284046+0000", 2021-01-11T04:37:12.615 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:54:53.213 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats , 2021-01-10T15:54:53.393 INFO:teuthology.orchestra.run.smithi044.stdout:60129542183 , 2021-01-10T15:54:53.394 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-01-11T04:37:12.616 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:04.324 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 , instructed osd(s) 6 to deep-scrub , 2021-01-10T15:55:04.650 INFO:tasks.ceph:Scrubbing osd.7 2021-01-11T04:37:12.617 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu2-testing-2021-01-10-1949/qa/tasks/ceph.py", line 1883, in task , yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 176, in run_tasks 2021-01-11T04:37:12.617 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:55:50.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.Qk6NhAVPxy , 2021-01-10T15:55:50.549 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.Qk6NhAVPxy is 947KB 2021-01-11T04:37:12.618 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-10T15:57:00.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-01-10T15:57:00.301 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-01-10T15:57:00.301 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-01-11T04:37:16.482 INFO:scrape:Found 14 distinct failure reasons 2021-01-11T04:37:16.484 INFO:scrape:Failure: failed to reach quorum size 9 before timeout expired 2021-01-11T04:37:16.484 INFO:scrape:1 jobs: ['5772996'] 2021-01-11T04:37:16.485 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{rhel_8}', 'thrashers/force-sync-many', 'workloads/snaps-few-objects}'] 2021-01-11T04:37:16.485 INFO:scrape: 2021-01-11T04:37:16.486 INFO:scrape:Dead: b'2021-01-11T02:46:35.465 DEBUG:teuthology.exit:Finished running handlers' 2021-01-11T04:37:16.486 INFO:scrape:9 jobs: ['5773059', '5772990', '5773031', '5773433', '5773331', '5773132', '5773149', '5773359', '5773107'] 2021-01-11T04:37:16.487 INFO:scrape:suites intersection: [] 2021-01-11T04:37:16.487 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/objectstore/{backends/objectstore', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}', 'workloads/radosbench}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/small-objects-localized}', 'workloads/snaps-few-objects}'] 2021-01-11T04:37:16.487 INFO:scrape: 2021-01-11T04:37:16.488 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2021-01-11T04:37:16.488 INFO:scrape:2 jobs: ['5773271', '5773110'] 2021-01-11T04:37:16.489 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2021-01-11T04:37:16.489 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2021-01-11T04:37:16.490 INFO:scrape: 2021-01-11T04:37:16.490 INFO:scrape:Failure: Command failed on smithi037 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:b0f788f13cdd55cc878274393b523e20cb3d36b0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 770e1898-5357-11eb-8f8e-001a4aab830c -- ceph orch host add smithi037' 2021-01-11T04:37:16.490 INFO:scrape:1 jobs: ['5773094'] 2021-01-11T04:37:16.491 INFO:scrape:suites: ['fixed-2', 'mode/packaged', 'mon_election/classic', 'msgr/async-v2only', 'rados/cephadm/with-work/{distro/centos_8.0', 'start', 'tasks/rados_api_tests}'] 2021-01-11T04:37:16.491 INFO:scrape: 2021-01-11T04:37:16.491 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-snaps.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh' 2021-01-11T04:37:16.492 INFO:scrape:2 jobs: ['5773391', '5773165'] 2021-01-11T04:37:16.492 INFO:scrape:suites intersection: ['workloads/scrub}'] 2021-01-11T04:37:16.493 INFO:scrape:suites union: ['rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/scrub}'] 2021-01-11T04:37:16.493 INFO:scrape: 2021-01-11T04:37:16.494 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-01-11T04:37:16.494 INFO:scrape:2 jobs: ['5773404', '5773151'] 2021-01-11T04:37:16.494 INFO:scrape:suites intersection: ['msgr/async-v2only', 'rados'] 2021-01-11T04:37:16.495 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'thrashers/default', 'thrashosds-health', 'workloads/redirect_set_object}'] 2021-01-11T04:37:16.495 INFO:scrape: 2021-01-11T04:37:16.496 INFO:scrape:Failure: Command failed on smithi149 with status 1: 'sudo docker kill -s 1 ceph-23ebf232-535e-11eb-8f8e-001a4aab830c-osd.8' 2021-01-11T04:37:16.496 INFO:scrape:1 jobs: ['5773185'] 2021-01-11T04:37:16.496 INFO:scrape:suites: ['1-install/luminous-v1only', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] 2021-01-11T04:37:16.497 INFO:scrape: 2021-01-11T04:37:16.497 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2021-01-11T04:37:16.498 INFO:scrape:2 jobs: ['5773337', '5773074'] 2021-01-11T04:37:16.498 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2021-01-11T04:37:16.499 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2021-01-11T04:37:16.499 INFO:scrape: 2021-01-11T04:37:16.499 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2021-01-11T04:37:16.500 INFO:scrape:5 jobs: ['5773225', '5773233', '5773381', '5773268', '5773105'] 2021-01-11T04:37:16.500 INFO:scrape:suites intersection: [] 2021-01-11T04:37:16.501 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'distro$/{centos_8.0}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2021-01-11T04:37:16.501 INFO:scrape: 2021-01-11T04:37:16.501 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2021-01-11T04:37:16.502 INFO:scrape:3 jobs: ['5773406', '5773255', '5773081'] 2021-01-11T04:37:16.502 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health'] 2021-01-11T04:37:16.503 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-snaps}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects-localized}'] 2021-01-11T04:37:16.503 INFO:scrape: 2021-01-11T04:37:16.503 INFO:scrape:Timeout 120 running ceph --cluster ceph quorum_status 2021-01-11T04:37:16.504 INFO:scrape:1 jobs: ['5773043'] 2021-01-11T04:37:16.504 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/many', 'msgr/async', 'no_pools', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/multimon/{clusters/21', 'supported-random-distro$/{rhel_8}', 'tasks/mon_recovery}'] 2021-01-11T04:37:16.505 INFO:scrape: 2021-01-11T04:37:16.505 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2021-01-11T04:37:16.505 INFO:scrape:1 jobs: ['5773050'] 2021-01-11T04:37:16.506 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests}'] 2021-01-11T04:37:16.506 INFO:scrape: 2021-01-11T04:37:16.507 INFO:scrape:Crash: "2021-01-10T17:51:44.498834+0000 mon.b (mon.0) 1112 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log 2021-01-11T04:37:16.507 INFO:scrape:ceph version 16.0.0-8845-gb0f788f1 (b0f788f13cdd55cc878274393b523e20cb3d36b0) pacific (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f62505f0980] 2: [0x557c41d65520] 2021-01-11T04:37:16.507 INFO:scrape:1 jobs: ['5773377'] 2021-01-11T04:37:16.508 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}'] 2021-01-11T04:37:16.508 INFO:scrape: 2021-01-11T04:37:16.509 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2021-01-11T04:37:16.509 INFO:scrape:1 jobs: ['5773307'] 2021-01-11T04:37:16.509 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'rados', 'rados/singleton/{all/rebuild-mondb', 'supported-random-distro$/{rhel_8}}'] 2021-01-11T04:37:16.510 INFO:scrape: 2021-01-11T04:37:16.525 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/ failed: 23 dead: 9 running: 0 waiting: 0 queued: 0 passed: 414 Fail ================================================================= [5772996] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:30:34 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772996/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772996/ failed to reach quorum size 9 before timeout expired [5773043] rados/multimon/{clusters/21 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:15:08 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773043/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773043/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e15396fe2f1845f1a52ee173332f81d0 Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status' [5773050] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:22:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773050/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773050/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2d96e2c8e8c4152989dc0c944d3f09a Command failed (workunit test rados/test.sh) on smithi081 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5773074] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:46:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773074/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773074/ sentry: https://sentry.ceph.com/organizations/ceph/?query=892c8ae4c851483cabdba931c2250d22 Command failed (workunit test rados/test.sh) on smithi080 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5773081] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 05:59:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773081/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773081/ Command failed on smithi148 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5773094] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:43:10 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773094/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773094/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4aea0e18987346cc83fffe911c1460e5 Command failed on smithi037 with status 1: 'sudo cephadm --image quay.ceph.io/ceph-ci/ceph:b0f788f13cdd55cc878274393b523e20cb3d36b0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 770e1898-5357-11eb-8f8e-001a4aab830c -- ceph orch host add smithi037' [5773105] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:17:41 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773105/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9f0349d43b04718ad313e62a2c0d6fd reached maximum tries (180) after waiting for 180 seconds [5773110] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:07:22 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773110/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773110/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7d63b40cd9245f78a772e4d351061be Command failed (workunit test cephadm/test_adoption.sh) on smithi074 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5773151] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:13:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773151/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773151/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op copy_from 100 --op write_excl 50 --pool unique_pool_0' [5773165] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/scrub} ----------------------------------------------------------------- time: 01:13:57 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773165/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773165/ sentry: https://sentry.ceph.com/organizations/ceph/?query=881ecb552de54d559f97da41ec9c889d Command failed (workunit test scrub/osd-scrub-snaps.sh) on smithi091 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- snaps.sh' [5773185] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:34:44 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773185/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773185/ Command failed on smithi149 with status 1: 'sudo docker kill -s 1 ceph-23ebf232-535e-11eb-8f8e-001a4aab830c-osd.8' [5773225] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:16:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773225/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773225/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7dd0c5e6984747debdb96ce9fcf28b0f reached maximum tries (180) after waiting for 180 seconds [5773233] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 06:57:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773233/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773233/ sentry: https://sentry.ceph.com/organizations/ceph/?query=692efe60dfc44daea37dc5009d668823 reached maximum tries (500) after waiting for 3000 seconds [5773255] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 09:43:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773255/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773255/ Command failed on smithi165 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [5773268] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:18:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773268/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773268/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50ef185273b84b50ab19862a922deeae reached maximum tries (180) after waiting for 180 seconds [5773271] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:07:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773271/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773271/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9e9b6a8f75645b9acdc7696e6e042bb Command failed (workunit test cephadm/test_adoption.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5773307] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:17:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773307/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773307/ Command failed on smithi103 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [5773337] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:41:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773337/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773337/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d1683ab71664b62b2fabee4b23dfb2f Command failed (workunit test rados/test.sh) on smithi133 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5773377] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:23:27 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773377/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773377/ "2021-01-10T17:51:44.498834+0000 mon.b (mon.0) 1112 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log [5773381] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773381/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61c09e487dc9479986d9d5220e716d98 reached maximum tries (180) after waiting for 180 seconds [5773391] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/scrub} ----------------------------------------------------------------- time: 01:12:33 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773391/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773391/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7d3a62c4036f477f9e24c60886fa17c3 Command failed (workunit test scrub/osd-scrub-snaps.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0f788f13cdd55cc878274393b523e20cb3d36b0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub- snaps.sh' [5773404] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:13:51 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773404/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773404/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5773406] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 04:16:53 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773406/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773406/ Command failed on smithi086 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Dead ================================================================= [5772990] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772990/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772990/ [5773031] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773031/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773031/ [5773059] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773059/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773059/ [5773107] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773107/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773107/ [5773132] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773132/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773132/ [5773149] rados/objectstore/{backends/objectstore supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773149/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773149/ [5773331] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773331/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773331/ [5773359] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773359/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773359/ [5773433] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773433/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773433/ Pass ================================================================= [5772989] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:27 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772989/ [5772991] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772991/ [5772992] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:13:28 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772992/ [5772993] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772993/ [5772994] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:07:31 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772994/ [5772995] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic} time: 00:18:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772995/ [5772997] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:05:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772997/ [5772998] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:08:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772998/ [5772999] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:25:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5772999/ [5773000] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:23:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773000/ [5773001] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:08:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773001/ [5773002] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773002/ [5773003] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}} time: 00:08:09 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773003/ [5773004] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773004/ [5773005] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big} time: 00:37:29 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773005/ [5773006] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:15:07 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773006/ [5773007] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/failover} time: 00:14:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773007/ [5773008] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} time: 00:15:09 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773008/ [5773009] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:37:35 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773009/ [5773010] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:22:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773010/ [5773011] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:09:33 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773011/ [5773012] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:29:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773012/ [5773013] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:12:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773013/ [5773014] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:11:31 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773014/ [5773015] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773015/ [5773016] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps} time: 00:17:09 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773016/ [5773017] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:29:27 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773017/ [5773018] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:20:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773018/ [5773019] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:28:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773019/ [5773020] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:06:36 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773020/ [5773021] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:06:35 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773021/ [5773022] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:22:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773022/ [5773023] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:43 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773023/ [5773024] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:27:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773024/ [5773025] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773025/ [5773026] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} time: 00:09:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773026/ [5773027] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:08:51 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773027/ [5773028] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773028/ [5773029] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773029/ [5773030] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/mon_recovery validater/lockdep} time: 00:12:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773030/ [5773032] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:31:30 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773032/ [5773033] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:06:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773033/ [5773034] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:28:15 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773034/ [5773035] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773035/ [5773036] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} time: 01:28:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773036/ [5773037] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:08:23 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773037/ [5773038] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:08:27 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773038/ [5773039] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:36:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773039/ [5773040] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773040/ [5773041] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/many workloads/pool-create-delete} time: 00:19:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773041/ [5773042] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:26:05 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773042/ [5773044] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773044/ [5773045] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:49:35 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773045/ [5773046] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/insights} time: 00:13:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773046/ [5773047] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/connectivity} time: 00:19:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773047/ [5773048] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:10:52 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773048/ [5773049] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:12:12 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773049/ [5773051] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:07:35 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773051/ [5773052] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{rhel_8}} time: 00:14:03 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773052/ [5773053] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:19:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773053/ [5773054] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:11:52 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773054/ [5773055] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:16:05 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773055/ [5773056] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:12:29 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773056/ [5773057] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:09:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773057/ [5773058] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:05 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773058/ [5773060] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:33:38 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773060/ [5773061] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:12:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773061/ [5773062] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:19:07 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773062/ [5773063] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:16:22 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773063/ [5773064] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_orch_cli} time: 00:16:38 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773064/ [5773065] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773065/ [5773066] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:21:34 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773066/ [5773067] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:20:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773067/ [5773068] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:25:07 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773068/ [5773069] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:08:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773069/ [5773070] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:22:29 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773070/ [5773071] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:12:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773071/ [5773072] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773072/ [5773073] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773073/ [5773075] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:31:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773075/ [5773076] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:33:58 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773076/ [5773077] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773077/ [5773078] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773078/ [5773079] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773079/ [5773080] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773080/ [5773082] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:26:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773082/ [5773083] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:11:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773083/ [5773084] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_5925} time: 00:09:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773084/ [5773085] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:47:51 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773085/ [5773086] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:23:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773086/ [5773087] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:29:37 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773087/ [5773088] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:08 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773088/ [5773089] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:16:10 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773089/ [5773090] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:29:02 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773090/ [5773091] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:11:33 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773091/ [5773092] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:24:57 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773092/ [5773093] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773093/ [5773095] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:28 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773095/ [5773096] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:38:53 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773096/ [5773097] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:15:03 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773097/ [5773098] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:53:10 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773098/ [5773099] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:27:43 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773099/ [5773100] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773100/ [5773101] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:26:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773101/ [5773102] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773102/ [5773103] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:14:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773103/ [5773104] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mon} time: 00:40:31 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773104/ [5773106] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773106/ [5773108] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:22:15 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773108/ [5773109] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:08:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773109/ [5773111] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:25:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773111/ [5773112] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:12:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773112/ [5773113] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:20:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773113/ [5773114] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773114/ [5773115] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:25:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773115/ [5773116] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:31 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773116/ [5773117] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_cls_all validater/lockdep} time: 00:18:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773117/ [5773118] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:14:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773118/ [5773119] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:09:28 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773119/ [5773120] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:45 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773120/ [5773121] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache} time: 00:12:22 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773121/ [5773122] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:30:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773122/ [5773123] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:12:07 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773123/ [5773124] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773124/ [5773125] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:33 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773125/ [5773126] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773126/ [5773127] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/rados_api_tests} time: 00:23:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773127/ [5773128] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench} time: 02:22:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773128/ [5773129] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:21:52 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773129/ [5773130] rados/multimon/{clusters/6 mon_election/classic msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:04:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773130/ [5773131] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/progress} time: 00:19:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773131/ [5773133] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773133/ [5773134] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:23:45 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773134/ [5773135] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773135/ [5773136] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:07:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773136/ [5773137] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 01:12:23 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773137/ [5773138] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:26:15 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773138/ [5773139] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:40:05 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773139/ [5773140] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/radosbench} time: 02:16:07 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773140/ [5773141] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:14:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773141/ [5773142] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:14:08 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773142/ [5773143] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:18:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773143/ [5773144] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:12:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773144/ [5773145] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:10:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773145/ [5773146] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:21:02 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773146/ [5773147] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:14:51 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773147/ [5773148] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773148/ [5773150] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:16:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773150/ [5773152] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/connectivity} time: 00:17:10 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773152/ [5773153] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:12:43 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773153/ [5773154] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:07:25 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773154/ [5773155] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:52 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773155/ [5773156] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:36:20 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773156/ [5773157] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:24:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773157/ [5773158] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch} time: 00:16:36 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773158/ [5773159] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:45 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773159/ [5773160] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/mon_recovery validater/valgrind} time: 00:29:12 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773160/ [5773161] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:09:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773161/ [5773162] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773162/ [5773163] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:20:31 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773163/ [5773164] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:20:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773164/ [5773166] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:11:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773166/ [5773167] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:22:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773167/ [5773168] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773168/ [5773169] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:47:36 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773169/ [5773170] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:14:43 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773170/ [5773171] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:24:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773171/ [5773172] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:33:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773172/ [5773173] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:12:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773173/ [5773174] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:33:29 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773174/ [5773175] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:51 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773175/ [5773176] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:08:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773176/ [5773177] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:13:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773177/ [5773178] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:10:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773178/ [5773179] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:30:51 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773179/ [5773180] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:24:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773180/ [5773181] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm_repos} time: 00:03:49 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773181/ [5773182] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:14:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773182/ [5773183] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:17 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773183/ [5773184] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:12:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773184/ [5773186] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:23:38 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773186/ [5773187] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:22:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773187/ [5773188] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773188/ [5773189] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:45:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773189/ [5773190] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:11:30 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773190/ [5773191] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:20:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773191/ [5773192] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:15:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773192/ [5773193] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:06:17 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773193/ [5773194] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773194/ [5773195] rados/objectstore/{backends/alloc-hint supported-random-distro$/{rhel_8}} time: 00:09:53 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773195/ [5773196] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:06:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773196/ [5773197] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:32:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773197/ [5773198] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:58 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773198/ [5773199] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/crush} time: 00:11:27 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773199/ [5773200] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} time: 05:25:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773200/ [5773201] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:20:41 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773201/ [5773202] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:17:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773202/ [5773203] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:27:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773203/ [5773204] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:18:24 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773204/ [5773205] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773205/ [5773206] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:22:15 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773206/ [5773207] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:37 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773207/ [5773208] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} time: 00:24:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773208/ [5773209] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:20:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773209/ [5773210] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_8}} time: 00:13:34 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773210/ [5773211] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:29:17 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773211/ [5773212] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:24:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773212/ [5773213] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:57 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773213/ [5773214] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773214/ [5773215] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773215/ [5773216] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:27:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773216/ [5773217] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:25:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773217/ [5773218] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:19:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773218/ [5773219] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:06:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773219/ [5773220] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:12:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773220/ [5773221] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:11:27 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773221/ [5773222] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:09:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773222/ [5773223] rados/multimon/{clusters/21 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:09:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773223/ [5773224] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:13:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773224/ [5773226] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:57 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773226/ [5773227] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773227/ [5773228] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:03 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773228/ [5773229] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:31:17 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773229/ [5773230] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rados_api_tests} time: 00:27:36 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773230/ [5773231] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 00:36:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773231/ [5773232] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:19:23 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773232/ [5773234] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:08:44 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773234/ [5773235] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:24:07 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773235/ [5773236] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:36:37 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773236/ [5773237] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:29:09 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773237/ [5773238] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:23 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773238/ [5773239] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect} time: 00:12:30 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773239/ [5773240] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:12:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773240/ [5773241] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:13:37 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773241/ [5773242] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:14:52 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773242/ [5773243] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:11:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773243/ [5773244] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773244/ [5773245] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:08:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773245/ [5773246] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773246/ [5773247] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:12:17 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773247/ [5773248] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:15:10 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773248/ [5773249] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773249/ [5773250] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773250/ [5773251] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/valgrind} time: 02:14:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773251/ [5773252] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:04:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773252/ [5773253] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/set-chunks-read} time: 00:16:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773253/ [5773254] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:41:02 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773254/ [5773256] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/many workloads/snaps-few-objects} time: 00:35:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773256/ [5773257] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:08:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773257/ [5773258] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:20:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773258/ [5773259] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:44 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773259/ [5773260] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:20:41 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773260/ [5773261] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:08:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773261/ [5773262] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:19:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773262/ [5773263] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mgr} time: 00:12:29 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773263/ [5773264] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:15 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773264/ [5773265] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:38 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773265/ [5773266] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:30 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773266/ [5773267] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects} time: 00:20:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773267/ [5773269] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:31:17 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773269/ [5773270] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:08:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773270/ [5773272] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:18:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773272/ [5773273] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773273/ [5773274] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} time: 00:31:09 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773274/ [5773275] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:25:14 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773275/ [5773276] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:08:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773276/ [5773277] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/rbd_cls} time: 00:19:22 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773277/ [5773278] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:33:05 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773278/ [5773279] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:41 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773279/ [5773280] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:11:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773280/ [5773281] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:36 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773281/ [5773282] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:14:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773282/ [5773283] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773283/ [5773284] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:45:17 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773284/ [5773285] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:21 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773285/ [5773286] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:21:45 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773286/ [5773287] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-big} time: 00:12:49 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773287/ [5773288] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773288/ [5773289] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:12:24 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773289/ [5773290] rados/singleton/{all/random-eio mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:25:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773290/ [5773291] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-small} time: 00:13:57 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773291/ [5773292] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/misc} time: 00:28:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773292/ [5773293] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773293/ [5773294] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/lockdep} time: 00:13:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773294/ [5773295] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:24 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773295/ [5773296] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}} time: 00:08:08 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773296/ [5773297] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:25:38 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773297/ [5773298] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:37:12 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773298/ [5773299] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/one workloads/pool-create-delete} time: 00:17:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773299/ [5773300] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects} time: 00:23:05 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773300/ [5773301] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:20:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773301/ [5773302] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} tasks/failover} time: 00:13:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773302/ [5773303] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773303/ [5773304] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps} time: 00:23:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773304/ [5773305] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:11:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773305/ [5773306] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:20 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773306/ [5773308] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:25:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773308/ [5773309] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:11:57 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773309/ [5773310] rados/multimon/{clusters/6 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:09:58 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773310/ [5773311] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:32:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773311/ [5773312] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:28:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773312/ [5773313] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:25:52 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773313/ [5773314] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:14:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773314/ [5773315] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache} time: 00:08:33 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773315/ [5773316] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:32:34 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773316/ [5773317] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:29 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773317/ [5773318] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:31:47 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773318/ [5773319] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:27:24 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773319/ [5773320] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/test_rbd_api} time: 00:18:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773320/ [5773321] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773321/ [5773322] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/rados_api_tests} time: 00:30:53 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773322/ [5773323] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:20 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773323/ [5773324] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:37:34 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773324/ [5773325] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 01:13:43 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773325/ [5773326] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773326/ [5773327] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} time: 00:16:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773327/ [5773328] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773328/ [5773329] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:02 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773329/ [5773330] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:09:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773330/ [5773332] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:28:43 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773332/ [5773333] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:14:01 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773333/ [5773334] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:28:34 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773334/ [5773335] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:20:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773335/ [5773336] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773336/ [5773338] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:12:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773338/ [5773339] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:10:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773339/ [5773340] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:14:27 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773340/ [5773341] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/rados_5925} time: 00:11:12 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773341/ [5773342] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:11:02 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773342/ [5773343] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:26:06 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773343/ [5773344] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:10:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773344/ [5773345] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:14:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773345/ [5773346] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773346/ [5773347] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:25:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773347/ [5773348] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/readwrite} time: 00:13:37 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773348/ [5773349] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:13 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773349/ [5773350] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:23:10 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773350/ [5773351] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:20:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773351/ [5773352] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:18:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773352/ [5773353] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm_repos} time: 00:03:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773353/ [5773354] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:09:20 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773354/ [5773355] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:26:37 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773355/ [5773356] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:19:53 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773356/ [5773357] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:23:08 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773357/ [5773358] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:15:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773358/ [5773360] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773360/ [5773361] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:05:52 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773361/ [5773362] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:18:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773362/ [5773363] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench} time: 00:59:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773363/ [5773364] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:21:15 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773364/ [5773365] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:12:12 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773365/ [5773366] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 01:19:49 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773366/ [5773367] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:23:44 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773367/ [5773368] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:18:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773368/ [5773369] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:16:22 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773369/ [5773370] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773370/ [5773371] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:08 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773371/ [5773372] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:29:41 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773372/ [5773373] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:33:03 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773373/ [5773374] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773374/ [5773375] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:19:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773375/ [5773376] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:28:30 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773376/ [5773378] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:53 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773378/ [5773379] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773379/ [5773380] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_cls_all validater/lockdep} time: 00:17:20 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773380/ [5773382] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/repair_test} time: 00:16:23 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773382/ [5773383] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:07:59 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773383/ [5773384] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:32:36 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773384/ [5773385] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_api_tests} time: 00:27:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773385/ [5773386] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:20:49 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773386/ [5773387] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:28:50 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773387/ [5773388] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:11:24 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773388/ [5773389] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:34 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773389/ [5773390] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:12:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773390/ [5773392] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:19:33 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773392/ [5773393] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:20:03 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773393/ [5773394] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:22:46 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773394/ [5773395] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:36:44 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773395/ [5773396] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:30 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773396/ [5773397] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:34:51 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773397/ [5773398] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:25 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773398/ [5773399] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:07:56 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773399/ [5773400] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps} time: 00:25:38 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773400/ [5773401] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:30:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773401/ [5773402] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:12:20 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773402/ [5773403] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-snaps-balanced} time: 00:22:58 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773403/ [5773405] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:10:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773405/ [5773407] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:26 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773407/ [5773408] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:20:07 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773408/ [5773409] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache} time: 00:13:00 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773409/ [5773410] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:10:55 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773410/ [5773411] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:07:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773411/ [5773412] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:11:39 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773412/ [5773413] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:24:42 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773413/ [5773414] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:09:58 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773414/ [5773415] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/rados_api_tests} time: 00:27:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773415/ [5773416] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:18:16 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773416/ [5773417] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:12:11 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773417/ [5773418] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:25 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773418/ [5773419] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:28:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773419/ [5773420] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:54 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773420/ [5773421] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/crush} time: 00:11:25 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773421/ [5773422] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} time: 04:51:48 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773422/ [5773423] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:53:25 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773423/ [5773424] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:18:18 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773424/ [5773425] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:33 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773425/ [5773426] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/valgrind} time: 02:10:37 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773426/ [5773427] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench} time: 01:27:19 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773427/ [5773428] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_8}} time: 00:13:04 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773428/ [5773429] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/progress} time: 00:18:32 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773429/ [5773430] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:43:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773430/ [5773431] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:08:03 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773431/ [5773432] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:26:40 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773432/ [5773434] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:38 info: http://pulpito.ceph.com/kchai-2021-01-10_14:19:31-rados-wip-kefu2-testing-2021-01-10-1949-distro-basic-smithi/5773434/