Found 364 jobs Ignoring malparsed backtrace: ceph version 16.0.0-9128-g9280858e (9280858e9fc1bfb270256febe37c17d78ff0e138) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f2abe7ba27a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x5576f6cdaf36] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-17T02:22:45.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-17T02:22:46.181 INFO:teuthology.orchestra.run.smithi154.stdout: Ignoring malparsed backtrace: 2021-01-17T02:23:47.984 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-17T02:23:48.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s , 2021-01-17T02:23:48.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s Ignoring malparsed backtrace: 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== , 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:^- hv01.front.sepia.ceph.com 3 6 377 8 -6603us[-6603us] +/- 128ms Ignoring malparsed backtrace: ceph version 16.0.0-9128-g9280858e (9280858e9fc1bfb270256febe37c17d78ff0e138) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f2abe7ba27a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x5576f6cdaf36] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-17T02:22:45.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-17T02:22:46.181 INFO:teuthology.orchestra.run.smithi154.stdout: Ignoring malparsed backtrace: 2021-01-17T02:23:47.984 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-17T02:23:48.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s , 2021-01-17T02:23:48.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s Ignoring malparsed backtrace: 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== , 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:^- hv01.front.sepia.ceph.com 3 6 377 8 -6603us[-6603us] +/- 128ms Ignoring malparsed backtrace: ceph version 16.0.0-9128-g9280858e (9280858e9fc1bfb270256febe37c17d78ff0e138) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f2abe7ba27a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x5576f6cdaf36] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-17T02:22:45.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-17T02:22:46.181 INFO:teuthology.orchestra.run.smithi154.stdout: Ignoring malparsed backtrace: 2021-01-17T02:23:47.984 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-17T02:23:48.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s , 2021-01-17T02:23:48.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s Ignoring malparsed backtrace: 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== , 2021-01-17T02:25:50.725 INFO:teuthology.orchestra.run.smithi196.stdout:^- hv01.front.sepia.ceph.com 3 6 377 8 -6603us[-6603us] +/- 128ms Ignoring malparsed backtrace: ceph version 16.0.0-9128-g9280858e (9280858e9fc1bfb270256febe37c17d78ff0e138) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7fd33674c03b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x56417d6b8c4c] Ignoring malparsed backtrace: 2021-01-16T19:50:12.886 INFO:teuthology.orchestra.run.smithi106.stdout: } , 2021-01-16T19:50:12.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:12.887 INFO:teuthology.orchestra.run.smithi106.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:12.907 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:12.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", , 2021-01-16T19:50:12.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:46:22.252356+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:12.929 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:12.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", , 2021-01-16T19:50:12.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:46:22.271792+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:12.949 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:12.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for blocked object", , 2021-01-16T19:50:12.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:46:22.272040+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:12.970 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:12.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", , 2021-01-16T19:50:12.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:46:22.516927+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:12.990 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ , 2021-01-16T19:50:12.990 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:12.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", Ignoring malparsed backtrace: 2021-01-16T19:50:13.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:13.011 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:13.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", Ignoring malparsed backtrace: 2021-01-16T19:50:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9138e-05 , 2021-01-16T19:50:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.060803656999999997 , 2021-01-16T19:50:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.248340845 , 2021-01-16T19:50:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: } , 2021-01-16T19:50:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: ] Ignoring malparsed backtrace: 2021-01-16T19:50:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:47:20.317959+0000", , 2021-01-16T19:50:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0950000000000001e-06 , 2021-01-16T19:50:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:47:30.247637+0000", , 2021-01-16T19:50:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00020710499999999999 , 2021-01-16T19:50:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.4616.0:5949 4.0 4:17cce13f:::smithi10620123-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e73)", Ignoring malparsed backtrace: 2021-01-16T19:50:13.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", , 2021-01-16T19:50:13.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:48:04.912374+0000", , 2021-01-16T19:50:13.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4294967295.9999728 Ignoring malparsed backtrace: 2021-01-16T19:50:13.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", , 2021-01-16T19:50:13.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:48:19.337453+0000", , 2021-01-16T19:50:13.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016635635999999999 Ignoring malparsed backtrace: 2021-01-16T19:50:13.568 INFO:teuthology.orchestra.run.smithi180.stdout: "events": [ , 2021-01-16T19:50:13.568 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:13.569 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "initiated", Ignoring malparsed backtrace: 2021-01-16T19:50:13.595 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:13.596 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:13.596 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "done", Ignoring malparsed backtrace: 2021-01-16T19:50:13.620 INFO:teuthology.orchestra.run.smithi180.stdout: "description": "osd_op(client.4616.0:3976 4.1 4:bc14c1b5:::smithi10620123-488:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e65)", , 2021-01-16T19:50:13.620 INFO:teuthology.orchestra.run.smithi180.stdout: "initiated_at": "2021-01-16T19:47:20.291802+0000", , 2021-01-16T19:50:13.620 INFO:teuthology.orchestra.run.smithi180.stdout: "age": 173.270897158, Ignoring malparsed backtrace: 2021-01-16T19:50:13.641 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 4294967295.9999475 , 2021-01-16T19:50:13.641 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:13.641 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:13.661 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.00010777600000000001 , 2021-01-16T19:50:13.661 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:13.661 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:13.681 INFO:teuthology.orchestra.run.smithi180.stdout: "type_data": { , 2021-01-16T19:50:13.681 INFO:teuthology.orchestra.run.smithi180.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-16T19:50:13.681 INFO:teuthology.orchestra.run.smithi180.stdout: "client_info": { Ignoring malparsed backtrace: 2021-01-16T19:50:13.701 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:20.779983+0000", , 2021-01-16T19:50:13.702 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 , 2021-01-16T19:50:13.702 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:13.722 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:20.780032+0000", , 2021-01-16T19:50:13.722 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 3.2311999999999998e-05 , 2021-01-16T19:50:13.722 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:13.742 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:20.799035+0000", , 2021-01-16T19:50:13.742 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 2.5522999999999999e-05 , 2021-01-16T19:50:13.743 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:13.763 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:21.396916+0000", , 2021-01-16T19:50:13.763 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 9.5456000000000002e-05 , 2021-01-16T19:50:13.763 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:13.783 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:21.451077+0000", , 2021-01-16T19:50:13.783 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 6.2853000000000003e-05 , 2021-01-16T19:50:13.783 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:13.804 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:21.450495+0000", , 2021-01-16T19:50:13.804 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.55319565999999998 , 2021-01-16T19:50:13.804 INFO:teuthology.orchestra.run.smithi180.stdout: } Ignoring malparsed backtrace: 2021-01-16T19:50:13.823 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.55253622899999999, , 2021-01-16T19:50:13.823 INFO:teuthology.orchestra.run.smithi180.stdout: "type_data": { , 2021-01-16T19:50:13.824 INFO:teuthology.orchestra.run.smithi180.stdout: "flag_point": "started", Ignoring malparsed backtrace: 2021-01-16T19:50:13.843 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "header_read", , 2021-01-16T19:50:13.843 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:20.899807+0000", , 2021-01-16T19:50:13.843 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.000182793 Ignoring malparsed backtrace: 2021-01-16T19:50:13.863 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "reached_pg", , 2021-01-16T19:50:13.863 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:20.919537+0000", , 2021-01-16T19:50:13.863 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 3.8550999999999997e-05 Ignoring malparsed backtrace: 2021-01-16T19:50:13.883 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "sub_op_commit_rec", , 2021-01-16T19:50:13.883 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:54.471608+0000", , 2021-01-16T19:50:13.883 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 4.8272999999999997e-05 Ignoring malparsed backtrace: 2021-01-16T19:50:13.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub , 2021-01-16T19:50:13.905 DEBUG:teuthology.orchestra.run.smithi180:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight , 2021-01-16T19:50:14.030 INFO:teuthology.orchestra.run.smithi180.stdout:{ Ignoring malparsed backtrace: 2021-01-16T19:50:14.304 INFO:teuthology.orchestra.run.smithi180.stdout: "age": 114.76788448800001, , 2021-01-16T19:50:14.304 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.66744777200000005, , 2021-01-16T19:50:14.305 INFO:teuthology.orchestra.run.smithi180.stdout: "type_data": { Ignoring malparsed backtrace: 2021-01-16T19:50:14.325 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:14.325 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "all_read", , 2021-01-16T19:50:14.325 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:19.522063+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:14.345 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:14.345 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "waiting for peered", , 2021-01-16T19:50:14.346 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:19.524143+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:14.365 INFO:teuthology.orchestra.run.smithi180.stdout: "client_info": { , 2021-01-16T19:50:14.365 INFO:teuthology.orchestra.run.smithi180.stdout: "client": "client.4616", , 2021-01-16T19:50:14.366 INFO:teuthology.orchestra.run.smithi180.stdout: "client_addr": "v1:172.21.15.106:0/1374983745", Ignoring malparsed backtrace: 2021-01-16T19:50:14.385 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:14.386 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:14.386 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2021-01-16T19:50:14.405 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:14.406 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:14.406 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "waiting for peered", Ignoring malparsed backtrace: 2021-01-16T19:50:14.425 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 , 2021-01-16T19:50:14.426 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:14.426 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:14.445 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 1.017754536 , 2021-01-16T19:50:14.446 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:14.446 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:14.465 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:53.406489+0000", , 2021-01-16T19:50:14.466 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 , 2021-01-16T19:50:14.466 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:14.485 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:53.407186+0000", , 2021-01-16T19:50:14.486 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 , 2021-01-16T19:50:14.486 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:14.505 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:53.407378+0000", , 2021-01-16T19:50:14.506 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 , 2021-01-16T19:50:14.506 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:14.525 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:53.407483+0000", , 2021-01-16T19:50:14.526 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.000110209 , 2021-01-16T19:50:14.526 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:14.545 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:54.444648+0000", , 2021-01-16T19:50:14.546 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.00015474600000000001 , 2021-01-16T19:50:14.546 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:14.565 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "initiated", , 2021-01-16T19:50:14.566 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:53.408041+0000", , 2021-01-16T19:50:14.566 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-01-16T19:50:14.585 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "header_read", , 2021-01-16T19:50:14.586 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:55.251327+0000", , 2021-01-16T19:50:14.586 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.00063341799999999996 Ignoring malparsed backtrace: 2021-01-16T19:50:14.992 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:57.298477+0000", , 2021-01-16T19:50:14.992 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 4294967295.999959 , 2021-01-16T19:50:14.993 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:15.014 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.111980371, , 2021-01-16T19:50:15.014 INFO:teuthology.orchestra.run.smithi180.stdout: "type_data": { , 2021-01-16T19:50:15.014 INFO:teuthology.orchestra.run.smithi180.stdout: "flag_point": "started", Ignoring malparsed backtrace: 2021-01-16T19:50:15.034 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "commit_sent", , 2021-01-16T19:50:15.035 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:57.817462+0000", , 2021-01-16T19:50:15.035 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 3.0521999999999997e-05 Ignoring malparsed backtrace: 2021-01-16T19:50:15.055 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.055 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "queued_for_pg", , 2021-01-16T19:50:15.055 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:58.032067+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:15.076 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:15.076 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.076 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "all_read", Ignoring malparsed backtrace: 2021-01-16T19:50:15.097 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 , 2021-01-16T19:50:15.097 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:15.098 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:15.118 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.119 INFO:teuthology.orchestra.run.smithi180.stdout: "description": "MOSDPGPush(3.0 118/115 [PushOp(3:34c28838:::smithi10620123-278:head, version: 92'730, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:34c28838:::smithi10620123-278:head@92'730, size: 3210434, copy_subset: [0~3210434], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", , 2021-01-16T19:50:15.119 INFO:teuthology.orchestra.run.smithi180.stdout: "initiated_at": "2021-01-16T19:49:01.326198+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:15.139 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:01.408481+0000", , 2021-01-16T19:50:15.140 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.02104172 , 2021-01-16T19:50:15.140 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:15.160 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "queued_for_pg", , 2021-01-16T19:50:15.161 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:01.484177+0000", , 2021-01-16T19:50:15.161 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0.091447351999999996 Ignoring malparsed backtrace: 2021-01-16T19:50:15.181 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.181 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "all_read", , 2021-01-16T19:50:15.181 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:01.492818+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:15.201 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.201 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "sub_op_commit_rec", , 2021-01-16T19:50:15.202 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:49:58.607599+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:15.222 INFO:teuthology.orchestra.run.smithi180.stdout: "events": [ , 2021-01-16T19:50:15.222 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.222 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "initiated", Ignoring malparsed backtrace: 2021-01-16T19:50:15.628 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.629 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "throttled", , 2021-01-16T19:50:15.629 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:25.601826+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:15.650 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.651 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "done", , 2021-01-16T19:50:15.651 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:31.221371+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:15.672 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:15.672 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.673 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2021-01-16T19:50:15.694 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 , 2021-01-16T19:50:15.694 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:15.694 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:15.716 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 8.7522999999999995e-05 , 2021-01-16T19:50:15.716 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:15.716 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:15.738 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:47:49.920922+0000", , 2021-01-16T19:50:15.738 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 1.7159999999999998e-05 , 2021-01-16T19:50:15.738 INFO:teuthology.orchestra.run.smithi180.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:15.760 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "initiated", , 2021-01-16T19:50:15.760 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:01.318532+0000", , 2021-01-16T19:50:15.760 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 0 Ignoring malparsed backtrace: 2021-01-16T19:50:15.781 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "commit_sent", , 2021-01-16T19:50:15.781 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:05.657945+0000", , 2021-01-16T19:50:15.782 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 9.2998999999999995e-05 Ignoring malparsed backtrace: 2021-01-16T19:50:15.803 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.803 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "dispatched", , 2021-01-16T19:50:15.803 INFO:teuthology.orchestra.run.smithi180.stdout: "time": "2021-01-16T19:48:14.927302+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:15.824 INFO:teuthology.orchestra.run.smithi180.stdout: "events": [ , 2021-01-16T19:50:15.824 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.824 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "initiated", Ignoring malparsed backtrace: 2021-01-16T19:50:15.845 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:15.846 INFO:teuthology.orchestra.run.smithi180.stdout: { , 2021-01-16T19:50:15.846 INFO:teuthology.orchestra.run.smithi180.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2021-01-16T19:50:15.867 INFO:teuthology.orchestra.run.smithi180.stdout: "duration": 1.9303000000000001e-05 , 2021-01-16T19:50:15.867 INFO:teuthology.orchestra.run.smithi180.stdout: }, , 2021-01-16T19:50:15.867 INFO:teuthology.orchestra.run.smithi180.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:16.020 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 , 2021-01-16T19:50:16.020 INFO:teuthology.orchestra.run.smithi106.stdout:} , 2021-01-16T19:50:16.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops Ignoring malparsed backtrace: 2021-01-16T19:50:16.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:16.311 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:16.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2021-01-16T19:50:16.331 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-01-16T19:50:16.331 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ , 2021-01-16T19:50:16.332 INFO:teuthology.orchestra.run.smithi106.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:16.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.078397142000000003 , 2021-01-16T19:50:16.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:16.352 INFO:teuthology.orchestra.run.smithi106.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:16.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:49:58.795692+0000", , 2021-01-16T19:50:16.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4815e-05 , 2021-01-16T19:50:16.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, Ignoring malparsed backtrace: 2021-01-16T19:50:16.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.029556529000000002, , 2021-01-16T19:50:16.396 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { , 2021-01-16T19:50:16.396 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2021-01-16T19:50:16.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", , 2021-01-16T19:50:16.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:50:03.827295+0000", , 2021-01-16T19:50:16.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054116963999999997 Ignoring malparsed backtrace: 2021-01-16T19:50:16.437 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:16.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", , 2021-01-16T19:50:16.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:50:04.157263+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:16.457 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2021-01-16T19:50:04.272783+0000", , 2021-01-16T19:50:16.458 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.01591099, , 2021-01-16T19:50:16.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037874996000000001, Ignoring malparsed backtrace: 2021-01-16T19:50:16.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:16.478 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:16.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", Ignoring malparsed backtrace: 2021-01-16T19:50:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089067400000000004 , 2021-01-16T19:50:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, , 2021-01-16T19:50:16.497 INFO:teuthology.orchestra.run.smithi106.stdout: { Ignoring malparsed backtrace: 2021-01-16T19:50:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: { , 2021-01-16T19:50:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(osd.3.13:1573 3.0 e158/151 3:3acbd10d:::smithi10620123-429:head v 158'758, mlcod=157'756)", , 2021-01-16T19:50:16.522 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2021-01-16T19:50:09.066191+0000", Ignoring malparsed backtrace: 2021-01-16T19:50:16.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2021-01-16T19:50:09.574584+0000", , 2021-01-16T19:50:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6737000000000002e-05 , 2021-01-16T19:50:16.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, Ignoring malparsed backtrace: set osd.2 primary-affinity to 1 (8655362) , 2021-01-16T19:50:39.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 , 2021-01-16T19:50:39.516+0000 7fd164e01700 -1 osd.2 181 handle_osd_map injecting map crc failure Ignoring malparsed backtrace: 2021-01-16T19:50:50.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 , instructed osd(s) 1 to deep-scrub , 2021-01-16T19:50:50.918 INFO:tasks.ceph:Scrubbing osd.2 Ignoring malparsed backtrace: 2021-01-16T19:51:25.549+0000 7f8c72136700 -1 osd.4 195 handle_osd_map injecting map crc failure , 2021-01-16T19:51:34.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-16T19:51:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: Ignoring malparsed backtrace: 2021-01-16T19:52:17.999 DEBUG:teuthology.orchestra.run.smithi180:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 , 2021-01-16T19:52:18.138 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi180.front.sepia.ceph.com , 2021-01-16T19:52:18.138 DEBUG:teuthology.orchestra.run.smithi180:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 Ignoring malparsed backtrace: 2021-01-16T19:53:19.835 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2021-01-16T19:53:19.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart , 2021-01-16T19:53:19.838 DEBUG:teuthology.orchestra.run.smithi180:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart Found 9 distinct failure reasons Dead: b'2021-01-17T14:19:59.717 DEBUG:teuthology.exit:Finished running handlers' 4 jobs: ['5792201', '5791951', '5792180', '5792133'] suites intersection: [] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/objectstore/{backends/objectstore', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2 jobs: ['5792176', '5791874'] suites intersection: ['mon_election/classic', 'rados'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/cache}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi053 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9280858e9fc1bfb270256febe37c17d78ff0e138 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2 jobs: ['5791950', '5792085'] suites intersection: ['task/test_cephadm}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] Failure: reached maximum tries (180) after waiting for 180 seconds 5 jobs: ['5792146', '5791956', '5791877', '5792092', '5791913'] suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_latest}', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi186 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9280858e9fc1bfb270256febe37c17d78ff0e138 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2 jobs: ['5792045', '5791866'] suites intersection: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] suites union: ['rados/cephadm/dashboard/{distro/centos_latest', 'task/test_e2e}'] Failure: Command failed (workunit test rados/test.sh) on smithi151 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9280858e9fc1bfb270256febe37c17d78ff0e138 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 5 jobs: ['5791994', '5791888', '5792104', '5792158', '5792191'] suites intersection: ['clusters/{fixed-2', 'msgr-failures/few', 'openstack}', 'rados'] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'd-thrash/default/{default', 'd-thrash/none', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests', 'tasks/rados_api_tests}', 'thrashers/default', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/rados_api_tests}'] Failure: failed to reach quorum size 9 before timeout expired 1 jobs: ['5792034'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/force-sync-many', 'workloads/rados_mon_osdmap_prune}'] Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9280858e9fc1bfb270256febe37c17d78ff0e138 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 1 jobs: ['5791917'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] Failure: "2021-01-16T20:20:17.160144+0000 mon.a (mon.0) 21 : cluster [WRN] Health check failed: 2/21 mons down, quorum a,c,b,d,f,e,g,i,h,j,l,k,m,n,p,r,q,s,t (MON_DOWN)" in cluster log 1 jobs: ['5791926'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/multimon/{clusters/21', 'supported-random-distro$/{rhel_8}', 'tasks/mon_clock_no_skews}']