Found 354 jobs Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Ignoring malparsed backtrace: ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f6d8f99e15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55eb603bff26] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-28T04:54:36.411 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-28T04:54:36.752 INFO:teuthology.orchestra.run.smithi073.stdout: Ignoring malparsed backtrace: 2021-01-28T04:56:19.792 INFO:tasks.ceph.osd.1:Stopped , 2021-01-28T04:56:19.793 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-01-28T04:56:19.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-01-28T05:04:42.923 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv01.front.sepia.ceph.com 3 6 377 54 -1761us[-1764us] +/- 104ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 2 6 377 50 +345us[ +342us] +/- 32ms , 2021-01-28T05:04:42.924 INFO:teuthology.orchestra.run.smithi073.stdout:^- hv03.front.sepia.ceph.com 3 6 377 52 +1625us[+1623us] +/- 122ms Ignoring malparsed backtrace: 2021-01-28T05:05:09.830 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262163 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest , 2021-01-28T05:05:09.844 INFO:teuthology.orchestra.run.smithi073.stdout: 262168 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jan 28 05:04 /home/ubuntu/cephtest/archive Found 13 distinct failure reasons Failure: reached maximum tries (180) after waiting for 180 seconds 5 jobs: ['5831436', '5831304', '5831394', '5831341', '5831488'] suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_8.0}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] Crash: Test failure: test_turn_off_module (tasks.mgr.test_progress.TestProgress) ceph version 16.1.0-8-g5f17c37f (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) 1: /lib64/libpthread.so.0(+0x12b20) [0x7fbf8c720b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55d8a09f37cb] 5: (KernelDevice::_aio_thread()+0x1254) [0x55d8a152abb4] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d8a1535eb1] 7: /lib64/libpthread.so.0(+0x814a) [0x7fbf8c71614a] 8: clone() 1 jobs: ['5831423'] suites: ['debug/mgr', 'mon_election/connectivity', 'objectstore/bluestore-stupid', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_8}', 'tasks/progress}'] Failure: Command failed on smithi080 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:5f17c37f78a331b7a4bf793890f9d324c64183e5 -v bootstrap --fsid 4d2e9f18-60d9-11eb-8f9c-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id a --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.80 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' 9 jobs: ['5831362', '5831636', '5831447', '5831308', '5831609', '5831619', '5831553', '5831432', '5831364'] suites intersection: [] suites union: ['fixed-2', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr/async', 'msgr/async-v2only', 'rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman', 'rados/cephadm/smoke/{distro/ubuntu_18.04_podman', 'rados/cephadm/with-work/{distro/ubuntu_18.04_podman', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'start', 'start}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}'] Failure: Command failed (workunit test rados/test.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5f17c37f78a331b7a4bf793890f9d324c64183e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2 jobs: ['5831420', '5831567'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] Failure: Command failed on smithi163 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'egrep \'"\'"\'(defer backfill|defer recovery)\'"\'"\' /var/log/ceph/ceph-osd.*.log\'' 1 jobs: ['5831527'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/recovery-preemption', 'supported-random-distro$/{ubuntu_latest}}'] Dead: b'2021-01-28T16:45:49.300 DEBUG:teuthology.exit:Finished running handlers' 3 jobs: ['5831618', '5831295', '5831541'] suites intersection: [] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/objectstore/{backends/objectstore', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{rhel_8}', 'thrashers/minsize_recovery', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-radosbench}', 'workloads/snaps-few-objects-balanced}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5f17c37f78a331b7a4bf793890f9d324c64183e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 3 jobs: ['5831565', '5831417', '5831293'] suites intersection: ['task/test_cephadm}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] Failure: "2021-01-28T05:42:26.405020+0000 osd.10 (osd.10) 50 : cluster [ERR] 5.17 deep-scrub : stat mismatch, got 4/4 objects, 0/0 clones, 0/0 dirty, 0/0 omap, 0/0 pinned, 4/4 hit_set_archive, 0/0 whiteouts, 6781/0 bytes, 0/0 manifest objects, 6781/6781 hit_set_archive bytes." in cluster log 1 jobs: ['5831474'] suites: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'mon_election/connectivity', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] Failure: Found coredumps on ubuntu@smithi150.front.sepia.ceph.com 2 jobs: ['5831397', '5831535'] suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] Failure: Command failed on smithi111 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:5f17c37f78a331b7a4bf793890f9d324c64183e5 -v bootstrap --fsid 04d0f89c-6106-11eb-8f9d-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.111:6789]' && sudo chmod +r /etc/ceph/ceph.client.admin.keyring" 1 jobs: ['5831452'] suites: ['fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async-v1only', 'rados/cephadm/with-work/{distro/ubuntu_18.04_podman', 'start', 'tasks/rados_api_tests}'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi114 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5f17c37f78a331b7a4bf793890f9d324c64183e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2 jobs: ['5831476', '5831294'] suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 1 jobs: ['5831602'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{rhel_8}}'] Failure: "2021-01-28T04:30:22.752461+0000 mon.a (mon.0) 145 : cluster [WRN] Health check failed: 71 slow ops, oldest one blocked for 30 sec, osd.0 has slow ops (SLOW_OPS)" in cluster log 1 jobs: ['5831561'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'supported-random-distro$/{rhel_8}}']