2021-01-09T20:27:50.619 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-09T20:27:50.625 INFO:teuthology.results:subset: '1/50000' 2021-01-09T20:27:50.625 INFO:teuthology.results:seed: '1422' 2021-01-09T23:32:53.004 INFO:root:teuthology version: 1.0.0-77817aa4 2021-01-09T23:32:53.044 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-10T11:14:17.866 INFO:teuthology.results:Tests finished! gathering results... 2021-01-10T11:15:05.959 INFO:scrape:Found 378 jobs 2021-01-10T11:15:07.717 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd1309bc980] , 2: gsignal() 2021-01-10T11:15:07.737 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd1309bc980] , 2: gsignal() 2021-01-10T11:15:07.758 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd1309bc980] , 2: gsignal() 2021-01-10T11:15:07.780 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd1309bc980] , 2: gsignal() 2021-01-10T11:15:07.798 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd1309bc980] , 2: gsignal() 2021-01-10T11:15:07.819 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd1309bc980] , 2: gsignal() 2021-01-10T11:15:07.836 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fd1309bc980] , 2: gsignal() 2021-01-10T11:15:17.584 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:17.586 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:17.587 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:17.588 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:17.696 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:17.697 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:17.698 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:17.699 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:17.806 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:17.808 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:17.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:17.809 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:17.917 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:17.918 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:17.919 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:17.920 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.032 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.034 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.035 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.035 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.142 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.143 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.144 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.144 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.252 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.253 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.254 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.255 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.361 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.363 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.363 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.364 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.479 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.480 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.481 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.482 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.592 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.594 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.595 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.595 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.703 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.705 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.705 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.706 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.816 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8788-g6ccab5eb (6ccab5ebf4adbc97dcef8905e0bd6730184a85f4) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f0a1ae5485a] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x558d24b69914] 2021-01-10T11:15:18.817 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-09T23:33:07.405 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-09T23:33:07.682 INFO:teuthology.orchestra.run.smithi025.stdout: 2021-01-10T11:15:18.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:34:09.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi025.front.sepia.ceph.com , 2021-01-09T23:34:09.636 DEBUG:teuthology.orchestra.run.smithi025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2021-01-09T23:34:09.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi025.front.sepia.ceph.com 2021-01-10T11:15:18.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-09T23:36:06.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer , 2021-01-09T23:36:06.277 INFO:teuthology.task.internal:Duration was 880.839915 seconds , 2021-01-09T23:36:06.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-01-10T11:15:18.973 INFO:scrape:Found 12 distinct failure reasons 2021-01-10T11:15:18.975 INFO:scrape:Dead: b'2021-01-10T09:58:56.196 DEBUG:teuthology.exit:Finished running handlers' 2021-01-10T11:15:18.975 INFO:scrape:5 jobs: ['5770419', '5770252', '5770408', '5770383', '5770565'] 2021-01-10T11:15:18.976 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'thrashosds-health'] 2021-01-10T11:15:18.976 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'recovery-overrides/{default}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/radosbench}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects}'] 2021-01-10T11:15:18.977 INFO:scrape: 2021-01-10T11:15:18.977 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2021-01-10T11:15:18.978 INFO:scrape:8 jobs: ['5770581', '5770357', '5770350', '5770447', '5770492', '5770260', '5770393', '5770302'] 2021-01-10T11:15:18.978 INFO:scrape:suites intersection: [] 2021-01-10T11:15:18.979 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity}', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/cephadm/upgrade/{1-start', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2021-01-10T11:15:18.979 INFO:scrape: 2021-01-10T11:15:18.979 INFO:scrape:Failure: "2021-01-09T23:25:27.525766+0000 osd.1 (osd.1) 54 : cluster [ERR] 208.15 deep-scrub : stat mismatch, got 3/3 objects, 2/2 clones, 3/3 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 1/1 whiteouts, 20/20 bytes, 2/1 manifest objects, 0/0 hit_set_archive bytes." in cluster log 2021-01-10T11:15:18.980 INFO:scrape:1 jobs: ['5770549'] 2021-01-10T11:15:18.980 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/rados_api_tests}'] 2021-01-10T11:15:18.981 INFO:scrape: 2021-01-10T11:15:18.981 INFO:scrape:Failure: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-819d044a-52d0-11eb-8f8e-001a4aab830c-osd.8' 2021-01-10T11:15:18.982 INFO:scrape:1 jobs: ['5770544'] 2021-01-10T11:15:18.982 INFO:scrape:suites: ['1-install/luminous', 'backoff/normal', 'ceph', 'clusters/{openstack', 'd-balancer/on', 'distro$/{ubuntu_18.04}', 'mon_election/connectivity', 'msgr-failures/fastclose', 'rados', 'rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/snaps-few-objects}'] 2021-01-10T11:15:18.983 INFO:scrape: 2021-01-10T11:15:18.983 INFO:scrape:Failure: Command failed (workunit test misc/ver-health.sh) on smithi017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6ccab5ebf4adbc97dcef8905e0bd6730184a85f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2021-01-10T11:15:18.984 INFO:scrape:1 jobs: ['5770511'] 2021-01-10T11:15:18.984 INFO:scrape:suites: ['rados/standalone/{mon_election/classic', 'supported-random-distro$/{centos_8}', 'workloads/misc}'] 2021-01-10T11:15:18.985 INFO:scrape: 2021-01-10T11:15:18.985 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-filestore 20 --debug-bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' 2021-01-10T11:15:18.985 INFO:scrape:1 jobs: ['5770577'] 2021-01-10T11:15:18.986 INFO:scrape:suites: ['rados/objectstore/{backends/objectstore', 'supported-random-distro$/{ubuntu_latest}}'] 2021-01-10T11:15:18.986 INFO:scrape: 2021-01-10T11:15:18.987 INFO:scrape:Failure: Command failed (workunit test scrub/osd-scrub-snaps.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6ccab5ebf4adbc97dcef8905e0bd6730184a85f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh' 2021-01-10T11:15:18.987 INFO:scrape:2 jobs: ['5770589', '5770402'] 2021-01-10T11:15:18.988 INFO:scrape:suites intersection: ['supported-random-distro$/{rhel_8}', 'workloads/scrub}'] 2021-01-10T11:15:18.988 INFO:scrape:suites union: ['rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{rhel_8}', 'workloads/scrub}'] 2021-01-10T11:15:18.989 INFO:scrape: 2021-01-10T11:15:18.989 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi098 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6ccab5ebf4adbc97dcef8905e0bd6730184a85f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2021-01-10T11:15:18.989 INFO:scrape:1 jobs: ['5770298'] 2021-01-10T11:15:18.990 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2021-01-10T11:15:18.990 INFO:scrape: 2021-01-10T11:15:18.991 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6ccab5ebf4adbc97dcef8905e0bd6730184a85f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2021-01-10T11:15:18.991 INFO:scrape:1 jobs: ['5770477'] 2021-01-10T11:15:18.992 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2021-01-10T11:15:18.992 INFO:scrape: 2021-01-10T11:15:18.993 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2021-01-10T11:15:18.993 INFO:scrape:2 jobs: ['5770272', '5770496'] 2021-01-10T11:15:18.994 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2021-01-10T11:15:18.994 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2021-01-10T11:15:18.995 INFO:scrape: 2021-01-10T11:15:18.995 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6ccab5ebf4adbc97dcef8905e0bd6730184a85f4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2021-01-10T11:15:18.995 INFO:scrape:1 jobs: ['5770344'] 2021-01-10T11:15:18.996 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] 2021-01-10T11:15:18.996 INFO:scrape: 2021-01-10T11:15:18.997 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-01-10T11:15:18.997 INFO:scrape:1 jobs: ['5770574'] 2021-01-10T11:15:18.998 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{rhel_8}}'] 2021-01-10T11:15:18.998 INFO:scrape: