2021-03-10T07:19:40.396 INFO:root:teuthology version: 1.1.0-b6eecfe8 2021-03-10T07:19:40.399 INFO:teuthology.results:subset: '990/99999' 2021-03-10T07:19:40.399 INFO:teuthology.results:seed: '2477' 2021-03-10T09:23:43.155 INFO:root:teuthology version: 1.1.0-b6eecfe8 2021-03-10T09:23:43.186 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-10T10:21:47.087 INFO:teuthology.results:Tests finished! gathering results... 2021-03-10T10:21:49.383 INFO:scrape:Found 24 jobs 2021-03-10T10:21:50.540 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1738-gc6bef90e (c6bef90e2f561208e1aebe76e03aaeb6fae2552e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f34b259e1d8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f34b259e3f2] 2021-03-10T10:21:50.541 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:24.403955+0000", , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.3828000000000002e-05 , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.542 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.054 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:47.808296+0000", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9970999999999999e-05 2021-03-10T10:21:50.542 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:25.992112+0000", 2021-03-10T10:21:50.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.093 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.001076+0000", 2021-03-10T10:21:50.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:50.544 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:50.544 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.151 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", 2021-03-10T10:21:50.545 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.173 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", 2021-03-10T10:21:50.545 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: ] , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:50.546 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.5498e-05 , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.547 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.000114467 , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.547 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 3219 , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "events": [ 2021-03-10T10:21:50.548 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:50.733429+0000", , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 7.2470000000000002e-06 , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.548 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.288 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:05:57.900991+0000", , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00017601100000000001 , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.549 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.696 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:50.549 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", 2021-03-10T10:21:50.550 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 5.3550000000000004e-06 , 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.735 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.551 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 9.3345999999999997e-05 , 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.754 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.551 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.772 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.670756+0000", , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 8.3639000000000003e-05 , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.552 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.791 INFO:teuthology.orchestra.run.gibba022.stdout: "client_addr": "172.21.2.121:0/1891828164", , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 2451 , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.552 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "reached_pg", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:54.421046+0000", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.0502e-05 2021-03-10T10:21:50.553 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:07.942287+0000", 2021-03-10T10:21:50.553 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:08.497210+0000", 2021-03-10T10:21:50.554 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "initiated_at": "2021-03-10T09:04:28.375241+0000", , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "age": 93.305483183999996, , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 4.5438502850000004, 2021-03-10T10:21:50.555 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:50.555 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "started", 2021-03-10T10:21:50.556 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "flag_point": "started", , 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "client_info": { , 2021-03-10T09:06:01.926 INFO:teuthology.orchestra.run.gibba022.stdout: "client": "client.4370", 2021-03-10T10:21:50.556 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.142+0000 7f31ba7df700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 46087) UID: 0 , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout:{ , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout: "ops": [], 2021-03-10T10:21:50.557 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.357 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:50.557 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.377 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00015792399999999999 , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.558 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.978389+0000", , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.4764999999999997e-05 , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.558 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:17.511406+0000", , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.49168687900000002 , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:50.559 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:18.215030+0000", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.7939000000000001e-05 2021-03-10T10:21:50.559 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:49.483593+0000", 2021-03-10T10:21:50.560 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", 2021-03-10T10:21:50.561 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.1550000000000004e-06 , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.561 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.36809437499999997 , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: ] 2021-03-10T10:21:50.562 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.779007+0000", , 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 2.0275000000000002e-05 , 2021-03-10T09:06:02.535 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.562 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:31.952403+0000", , 2021-03-10T09:06:02.554 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9063999999999999e-05 2021-03-10T10:21:50.563 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:08.896100+0000", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.006128568 2021-03-10T10:21:50.563 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "initiated", , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:20.948037+0000", 2021-03-10T10:21:50.564 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "waiting for peered", , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:26.742027+0000", 2021-03-10T10:21:50.564 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:50.565 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "dispatched", 2021-03-10T10:21:50.565 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "sub_op_committed", 2021-03-10T10:21:50.566 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.068 INFO:teuthology.orchestra.run.gibba021.stdout: } 2021-03-10T10:21:50.567 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: "events": [ , 2021-03-10T09:06:03.087 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.567 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.105 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 3.3759999999999999e-06 , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.568 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.124 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.8424000000000001e-05 , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.568 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.143 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.0018942270000000001 , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.569 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.162 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 6.8745000000000007e-05 , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.569 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.181 INFO:teuthology.orchestra.run.gibba021.stdout: "type_data": { , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "client_info": { 2021-03-10T10:21:50.570 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068876+0000", , 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.1307000000000001e-05 , 2021-03-10T09:06:03.201 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:50.570 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068948+0000", , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.666e-06 , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:50.571 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout:} 2021-03-10T10:21:50.572 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:Added osd 6 , 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:inning osd , 2021-03-10T09:06:14.806 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-03-10T10:21:50.575 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:38.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542220 got 60129542220 for osd.4 , 2021-03-10T09:06:38.912 DEBUG:teuthology.orchestra.run.gibba021:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2021-03-10T09:06:39.136 INFO:teuthology.orchestra.run.gibba021.stdout:674309865515 2021-03-10T10:21:50.576 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-03-10T09:06:35.787066+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=35, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.320 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2021-03-10T09:06:15.448747+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=15, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:50.578 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2021-03-10T09:06:31.414269+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=31, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2021-03-10T09:06:37.166077+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=37, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.607 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:50.578 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:47.243 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2021-03-10T09:07:47.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba021.front.sepia.ceph.com , 2021-03-10T09:07:47.326 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-03-10T10:21:50.579 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:=============================================================================== , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +1646us[+1609us] +/- 89ms , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^* hv02.front.sepia.ceph.com 2 6 377 6 -485us[ -522us] +/- 29ms 2021-03-10T10:21:50.838 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1738-gc6bef90e (c6bef90e2f561208e1aebe76e03aaeb6fae2552e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f34b259e1d8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f34b259e3f2] 2021-03-10T10:21:50.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:24.403955+0000", , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.3828000000000002e-05 , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.054 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:47.808296+0000", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9970999999999999e-05 2021-03-10T10:21:50.840 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:25.992112+0000", 2021-03-10T10:21:50.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.093 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.001076+0000", 2021-03-10T10:21:50.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:50.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:50.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.151 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", 2021-03-10T10:21:50.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.173 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", 2021-03-10T10:21:50.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: ] , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:50.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.5498e-05 , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.000114467 , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 3219 , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "events": [ 2021-03-10T10:21:50.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:50.733429+0000", , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 7.2470000000000002e-06 , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.288 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:05:57.900991+0000", , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00017601100000000001 , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.696 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:50.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", 2021-03-10T10:21:50.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 5.3550000000000004e-06 , 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.735 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 9.3345999999999997e-05 , 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.754 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.772 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.670756+0000", , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 8.3639000000000003e-05 , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.850 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.791 INFO:teuthology.orchestra.run.gibba022.stdout: "client_addr": "172.21.2.121:0/1891828164", , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 2451 , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "reached_pg", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:54.421046+0000", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.0502e-05 2021-03-10T10:21:50.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:07.942287+0000", 2021-03-10T10:21:50.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:08.497210+0000", 2021-03-10T10:21:50.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "initiated_at": "2021-03-10T09:04:28.375241+0000", , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "age": 93.305483183999996, , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 4.5438502850000004, 2021-03-10T10:21:50.853 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:50.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "started", 2021-03-10T10:21:50.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "flag_point": "started", , 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "client_info": { , 2021-03-10T09:06:01.926 INFO:teuthology.orchestra.run.gibba022.stdout: "client": "client.4370", 2021-03-10T10:21:50.855 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.142+0000 7f31ba7df700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 46087) UID: 0 , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout:{ , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout: "ops": [], 2021-03-10T10:21:50.855 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.357 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:50.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.377 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00015792399999999999 , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.978389+0000", , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.4764999999999997e-05 , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:17.511406+0000", , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.49168687900000002 , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:50.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:18.215030+0000", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.7939000000000001e-05 2021-03-10T10:21:50.858 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:49.483593+0000", 2021-03-10T10:21:50.859 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", 2021-03-10T10:21:50.859 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.1550000000000004e-06 , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:50.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.36809437499999997 , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: ] 2021-03-10T10:21:50.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.779007+0000", , 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 2.0275000000000002e-05 , 2021-03-10T09:06:02.535 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:50.861 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:31.952403+0000", , 2021-03-10T09:06:02.554 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9063999999999999e-05 2021-03-10T10:21:50.861 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:08.896100+0000", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.006128568 2021-03-10T10:21:50.862 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "initiated", , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:20.948037+0000", 2021-03-10T10:21:50.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "waiting for peered", , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:26.742027+0000", 2021-03-10T10:21:50.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:50.864 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "dispatched", 2021-03-10T10:21:50.865 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "sub_op_committed", 2021-03-10T10:21:50.865 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.068 INFO:teuthology.orchestra.run.gibba021.stdout: } 2021-03-10T10:21:50.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: "events": [ , 2021-03-10T09:06:03.087 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.105 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 3.3759999999999999e-06 , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.124 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.8424000000000001e-05 , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.143 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.0018942270000000001 , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.868 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.162 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 6.8745000000000007e-05 , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:50.869 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.181 INFO:teuthology.orchestra.run.gibba021.stdout: "type_data": { , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "client_info": { 2021-03-10T10:21:50.869 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068876+0000", , 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.1307000000000001e-05 , 2021-03-10T09:06:03.201 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:50.870 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068948+0000", , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.666e-06 , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:50.870 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout:} 2021-03-10T10:21:50.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:Added osd 6 , 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:inning osd , 2021-03-10T09:06:14.806 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-03-10T10:21:50.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:38.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542220 got 60129542220 for osd.4 , 2021-03-10T09:06:38.912 DEBUG:teuthology.orchestra.run.gibba021:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2021-03-10T09:06:39.136 INFO:teuthology.orchestra.run.gibba021.stdout:674309865515 2021-03-10T10:21:50.875 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-03-10T09:06:35.787066+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=35, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.320 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2021-03-10T09:06:15.448747+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=15, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:50.877 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2021-03-10T09:06:31.414269+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=31, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2021-03-10T09:06:37.166077+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=37, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.607 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:50.878 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:47.243 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2021-03-10T09:07:47.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba021.front.sepia.ceph.com , 2021-03-10T09:07:47.326 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-03-10T10:21:50.878 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:=============================================================================== , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +1646us[+1609us] +/- 89ms , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^* hv02.front.sepia.ceph.com 2 6 377 6 -485us[ -522us] +/- 29ms 2021-03-10T10:21:51.136 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1738-gc6bef90e (c6bef90e2f561208e1aebe76e03aaeb6fae2552e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f34b259e1d8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f34b259e3f2] 2021-03-10T10:21:51.137 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:24.403955+0000", , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.3828000000000002e-05 , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.137 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.054 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:47.808296+0000", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9970999999999999e-05 2021-03-10T10:21:51.138 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:25.992112+0000", 2021-03-10T10:21:51.139 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.093 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.001076+0000", 2021-03-10T10:21:51.139 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:51.140 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.141 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.151 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", 2021-03-10T10:21:51.141 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.173 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", 2021-03-10T10:21:51.142 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: ] , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:51.142 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.5498e-05 , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.143 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.000114467 , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.143 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 3219 , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "events": [ 2021-03-10T10:21:51.144 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:50.733429+0000", , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 7.2470000000000002e-06 , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.145 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.288 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:05:57.900991+0000", , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00017601100000000001 , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.145 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.696 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:51.146 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", 2021-03-10T10:21:51.146 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 5.3550000000000004e-06 , 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.735 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.147 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 9.3345999999999997e-05 , 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.754 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.148 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.772 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.670756+0000", , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 8.3639000000000003e-05 , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.148 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.791 INFO:teuthology.orchestra.run.gibba022.stdout: "client_addr": "172.21.2.121:0/1891828164", , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 2451 , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.149 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "reached_pg", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:54.421046+0000", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.0502e-05 2021-03-10T10:21:51.149 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:07.942287+0000", 2021-03-10T10:21:51.150 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:08.497210+0000", 2021-03-10T10:21:51.150 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "initiated_at": "2021-03-10T09:04:28.375241+0000", , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "age": 93.305483183999996, , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 4.5438502850000004, 2021-03-10T10:21:51.151 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.152 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "started", 2021-03-10T10:21:51.152 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "flag_point": "started", , 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "client_info": { , 2021-03-10T09:06:01.926 INFO:teuthology.orchestra.run.gibba022.stdout: "client": "client.4370", 2021-03-10T10:21:51.153 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.142+0000 7f31ba7df700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 46087) UID: 0 , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout:{ , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout: "ops": [], 2021-03-10T10:21:51.153 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.357 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.154 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.377 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00015792399999999999 , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.155 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.978389+0000", , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.4764999999999997e-05 , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.155 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:17.511406+0000", , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.49168687900000002 , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:51.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:18.215030+0000", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.7939000000000001e-05 2021-03-10T10:21:51.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:49.483593+0000", 2021-03-10T10:21:51.157 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", 2021-03-10T10:21:51.157 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.1550000000000004e-06 , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.158 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.36809437499999997 , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: ] 2021-03-10T10:21:51.159 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.779007+0000", , 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 2.0275000000000002e-05 , 2021-03-10T09:06:02.535 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.159 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:31.952403+0000", , 2021-03-10T09:06:02.554 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9063999999999999e-05 2021-03-10T10:21:51.160 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:08.896100+0000", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.006128568 2021-03-10T10:21:51.160 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "initiated", , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:20.948037+0000", 2021-03-10T10:21:51.161 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "waiting for peered", , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:26.742027+0000", 2021-03-10T10:21:51.162 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.162 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "dispatched", 2021-03-10T10:21:51.163 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "sub_op_committed", 2021-03-10T10:21:51.163 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.068 INFO:teuthology.orchestra.run.gibba021.stdout: } 2021-03-10T10:21:51.164 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: "events": [ , 2021-03-10T09:06:03.087 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.164 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.105 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 3.3759999999999999e-06 , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.165 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.124 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.8424000000000001e-05 , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.166 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.143 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.0018942270000000001 , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.166 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.162 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 6.8745000000000007e-05 , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.167 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.181 INFO:teuthology.orchestra.run.gibba021.stdout: "type_data": { , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "client_info": { 2021-03-10T10:21:51.167 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068876+0000", , 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.1307000000000001e-05 , 2021-03-10T09:06:03.201 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.168 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068948+0000", , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.666e-06 , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.169 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout:} 2021-03-10T10:21:51.169 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:Added osd 6 , 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:inning osd , 2021-03-10T09:06:14.806 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-03-10T10:21:51.172 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:38.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542220 got 60129542220 for osd.4 , 2021-03-10T09:06:38.912 DEBUG:teuthology.orchestra.run.gibba021:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2021-03-10T09:06:39.136 INFO:teuthology.orchestra.run.gibba021.stdout:674309865515 2021-03-10T10:21:51.174 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-03-10T09:06:35.787066+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=35, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.320 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2021-03-10T09:06:15.448747+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=15, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:51.175 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2021-03-10T09:06:31.414269+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=31, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2021-03-10T09:06:37.166077+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=37, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.607 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:51.176 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:47.243 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2021-03-10T09:07:47.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba021.front.sepia.ceph.com , 2021-03-10T09:07:47.326 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-03-10T10:21:51.177 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:=============================================================================== , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +1646us[+1609us] +/- 89ms , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^* hv02.front.sepia.ceph.com 2 6 377 6 -485us[ -522us] +/- 29ms 2021-03-10T10:21:51.429 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1738-gc6bef90e (c6bef90e2f561208e1aebe76e03aaeb6fae2552e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f34b259e1d8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f34b259e3f2] 2021-03-10T10:21:51.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:24.403955+0000", , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.3828000000000002e-05 , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.054 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:47.808296+0000", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9970999999999999e-05 2021-03-10T10:21:51.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:25.992112+0000", 2021-03-10T10:21:51.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.093 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.001076+0000", 2021-03-10T10:21:51.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:51.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.151 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", 2021-03-10T10:21:51.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.173 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", 2021-03-10T10:21:51.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: ] , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:51.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.5498e-05 , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.000114467 , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 3219 , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "events": [ 2021-03-10T10:21:51.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:50.733429+0000", , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 7.2470000000000002e-06 , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.288 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:05:57.900991+0000", , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00017601100000000001 , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.696 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:51.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", 2021-03-10T10:21:51.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 5.3550000000000004e-06 , 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.735 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 9.3345999999999997e-05 , 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.754 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.772 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.670756+0000", , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 8.3639000000000003e-05 , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.791 INFO:teuthology.orchestra.run.gibba022.stdout: "client_addr": "172.21.2.121:0/1891828164", , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 2451 , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "reached_pg", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:54.421046+0000", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.0502e-05 2021-03-10T10:21:51.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:07.942287+0000", 2021-03-10T10:21:51.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:08.497210+0000", 2021-03-10T10:21:51.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "initiated_at": "2021-03-10T09:04:28.375241+0000", , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "age": 93.305483183999996, , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 4.5438502850000004, 2021-03-10T10:21:51.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "started", 2021-03-10T10:21:51.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "flag_point": "started", , 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "client_info": { , 2021-03-10T09:06:01.926 INFO:teuthology.orchestra.run.gibba022.stdout: "client": "client.4370", 2021-03-10T10:21:51.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.142+0000 7f31ba7df700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 46087) UID: 0 , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout:{ , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout: "ops": [], 2021-03-10T10:21:51.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.357 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.377 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00015792399999999999 , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.978389+0000", , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.4764999999999997e-05 , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:17.511406+0000", , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.49168687900000002 , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:51.449 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:18.215030+0000", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.7939000000000001e-05 2021-03-10T10:21:51.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:49.483593+0000", 2021-03-10T10:21:51.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", 2021-03-10T10:21:51.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.1550000000000004e-06 , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.36809437499999997 , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: ] 2021-03-10T10:21:51.452 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.779007+0000", , 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 2.0275000000000002e-05 , 2021-03-10T09:06:02.535 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.452 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:31.952403+0000", , 2021-03-10T09:06:02.554 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9063999999999999e-05 2021-03-10T10:21:51.453 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:08.896100+0000", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.006128568 2021-03-10T10:21:51.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "initiated", , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:20.948037+0000", 2021-03-10T10:21:51.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "waiting for peered", , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:26.742027+0000", 2021-03-10T10:21:51.455 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.455 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "dispatched", 2021-03-10T10:21:51.456 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "sub_op_committed", 2021-03-10T10:21:51.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.068 INFO:teuthology.orchestra.run.gibba021.stdout: } 2021-03-10T10:21:51.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: "events": [ , 2021-03-10T09:06:03.087 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.458 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.105 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 3.3759999999999999e-06 , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.458 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.124 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.8424000000000001e-05 , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.459 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.143 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.0018942270000000001 , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.460 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.162 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 6.8745000000000007e-05 , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.460 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.181 INFO:teuthology.orchestra.run.gibba021.stdout: "type_data": { , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "client_info": { 2021-03-10T10:21:51.461 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068876+0000", , 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.1307000000000001e-05 , 2021-03-10T09:06:03.201 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.461 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068948+0000", , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.666e-06 , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.462 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout:} 2021-03-10T10:21:51.463 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:Added osd 6 , 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:inning osd , 2021-03-10T09:06:14.806 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-03-10T10:21:51.466 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:38.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542220 got 60129542220 for osd.4 , 2021-03-10T09:06:38.912 DEBUG:teuthology.orchestra.run.gibba021:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2021-03-10T09:06:39.136 INFO:teuthology.orchestra.run.gibba021.stdout:674309865515 2021-03-10T10:21:51.467 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-03-10T09:06:35.787066+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=35, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.320 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2021-03-10T09:06:15.448747+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=15, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:51.469 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2021-03-10T09:06:31.414269+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=31, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2021-03-10T09:06:37.166077+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=37, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.607 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:51.469 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:47.243 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2021-03-10T09:07:47.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba021.front.sepia.ceph.com , 2021-03-10T09:07:47.326 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-03-10T10:21:51.470 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:=============================================================================== , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +1646us[+1609us] +/- 89ms , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^* hv02.front.sepia.ceph.com 2 6 377 6 -485us[ -522us] +/- 29ms 2021-03-10T10:21:51.723 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1738-gc6bef90e (c6bef90e2f561208e1aebe76e03aaeb6fae2552e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f34b259e1d8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f34b259e3f2] 2021-03-10T10:21:51.724 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:24.403955+0000", , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.3828000000000002e-05 , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.725 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.054 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:47.808296+0000", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9970999999999999e-05 2021-03-10T10:21:51.725 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:25.992112+0000", 2021-03-10T10:21:51.726 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.093 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.001076+0000", 2021-03-10T10:21:51.726 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:51.727 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.728 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.151 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", 2021-03-10T10:21:51.728 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.173 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", 2021-03-10T10:21:51.729 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: ] , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:51.729 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.5498e-05 , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.730 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.000114467 , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.730 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 3219 , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "events": [ 2021-03-10T10:21:51.731 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:50.733429+0000", , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 7.2470000000000002e-06 , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.732 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.288 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:05:57.900991+0000", , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00017601100000000001 , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.732 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.696 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:51.733 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", 2021-03-10T10:21:51.733 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 5.3550000000000004e-06 , 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.735 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.734 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 9.3345999999999997e-05 , 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.754 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.734 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.772 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.670756+0000", , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 8.3639000000000003e-05 , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.735 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.791 INFO:teuthology.orchestra.run.gibba022.stdout: "client_addr": "172.21.2.121:0/1891828164", , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 2451 , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.736 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "reached_pg", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:54.421046+0000", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.0502e-05 2021-03-10T10:21:51.736 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:07.942287+0000", 2021-03-10T10:21:51.737 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:08.497210+0000", 2021-03-10T10:21:51.737 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "initiated_at": "2021-03-10T09:04:28.375241+0000", , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "age": 93.305483183999996, , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 4.5438502850000004, 2021-03-10T10:21:51.738 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.738 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "started", 2021-03-10T10:21:51.739 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "flag_point": "started", , 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "client_info": { , 2021-03-10T09:06:01.926 INFO:teuthology.orchestra.run.gibba022.stdout: "client": "client.4370", 2021-03-10T10:21:51.740 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.142+0000 7f31ba7df700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 46087) UID: 0 , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout:{ , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout: "ops": [], 2021-03-10T10:21:51.740 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.357 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:51.741 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.377 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00015792399999999999 , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.741 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.978389+0000", , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.4764999999999997e-05 , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.742 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:17.511406+0000", , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.49168687900000002 , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:51.742 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:18.215030+0000", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.7939000000000001e-05 2021-03-10T10:21:51.743 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:49.483593+0000", 2021-03-10T10:21:51.743 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", 2021-03-10T10:21:51.744 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.1550000000000004e-06 , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:51.745 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.36809437499999997 , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: ] 2021-03-10T10:21:51.745 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.779007+0000", , 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 2.0275000000000002e-05 , 2021-03-10T09:06:02.535 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:51.746 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:31.952403+0000", , 2021-03-10T09:06:02.554 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9063999999999999e-05 2021-03-10T10:21:51.746 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:08.896100+0000", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.006128568 2021-03-10T10:21:51.747 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "initiated", , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:20.948037+0000", 2021-03-10T10:21:51.747 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "waiting for peered", , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:26.742027+0000", 2021-03-10T10:21:51.748 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.749 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "dispatched", 2021-03-10T10:21:51.749 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "sub_op_committed", 2021-03-10T10:21:51.750 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.068 INFO:teuthology.orchestra.run.gibba021.stdout: } 2021-03-10T10:21:51.750 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: "events": [ , 2021-03-10T09:06:03.087 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.751 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.105 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 3.3759999999999999e-06 , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.751 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.124 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.8424000000000001e-05 , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.752 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.143 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.0018942270000000001 , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.753 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.162 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 6.8745000000000007e-05 , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:51.753 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.181 INFO:teuthology.orchestra.run.gibba021.stdout: "type_data": { , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "client_info": { 2021-03-10T10:21:51.754 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068876+0000", , 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.1307000000000001e-05 , 2021-03-10T09:06:03.201 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.754 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068948+0000", , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.666e-06 , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:51.755 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout:} 2021-03-10T10:21:51.756 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:Added osd 6 , 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:inning osd , 2021-03-10T09:06:14.806 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-03-10T10:21:51.759 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:38.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542220 got 60129542220 for osd.4 , 2021-03-10T09:06:38.912 DEBUG:teuthology.orchestra.run.gibba021:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2021-03-10T09:06:39.136 INFO:teuthology.orchestra.run.gibba021.stdout:674309865515 2021-03-10T10:21:51.760 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-03-10T09:06:35.787066+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=35, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.320 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2021-03-10T09:06:15.448747+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=15, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:51.762 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2021-03-10T09:06:31.414269+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=31, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2021-03-10T09:06:37.166077+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=37, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.607 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:51.762 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:47.243 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2021-03-10T09:07:47.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba021.front.sepia.ceph.com , 2021-03-10T09:07:47.326 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-03-10T10:21:51.763 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:=============================================================================== , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +1646us[+1609us] +/- 89ms , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^* hv02.front.sepia.ceph.com 2 6 377 6 -485us[ -522us] +/- 29ms 2021-03-10T10:21:52.016 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1738-gc6bef90e (c6bef90e2f561208e1aebe76e03aaeb6fae2552e) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f34b259e1d8] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f34b259e3f2] 2021-03-10T10:21:52.017 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:24.403955+0000", , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.3828000000000002e-05 , 2021-03-10T09:06:01.035 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:52.018 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.054 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:02:47.808296+0000", , 2021-03-10T09:06:01.055 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9970999999999999e-05 2021-03-10T10:21:52.019 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.074 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:25.992112+0000", 2021-03-10T10:21:52.019 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.093 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", , 2021-03-10T09:06:01.094 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.001076+0000", 2021-03-10T10:21:52.020 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.113 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:52.020 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.132 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:52.021 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.151 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.152 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", 2021-03-10T10:21:52.021 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.172 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.173 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "waiting for peered", 2021-03-10T10:21:52.022 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: ] , 2021-03-10T09:06:01.192 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:52.023 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.5498e-05 , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.211 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:52.023 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.000114467 , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.230 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:52.024 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 3219 , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.250 INFO:teuthology.orchestra.run.gibba022.stdout: "events": [ 2021-03-10T10:21:52.024 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:50.733429+0000", , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 7.2470000000000002e-06 , 2021-03-10T09:06:01.269 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:52.025 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.288 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:05:57.900991+0000", , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00017601100000000001 , 2021-03-10T09:06:01.289 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:52.025 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.695 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.696 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", 2021-03-10T10:21:52.026 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.715 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", 2021-03-10T10:21:52.027 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 5.3550000000000004e-06 , 2021-03-10T09:06:01.734 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.735 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:52.027 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 9.3345999999999997e-05 , 2021-03-10T09:06:01.753 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.754 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:52.028 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.772 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.670756+0000", , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 8.3639000000000003e-05 , 2021-03-10T09:06:01.773 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:52.028 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.791 INFO:teuthology.orchestra.run.gibba022.stdout: "client_addr": "172.21.2.121:0/1891828164", , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: "tid": 2451 , 2021-03-10T09:06:01.792 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:52.029 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "reached_pg", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:54.421046+0000", , 2021-03-10T09:06:01.811 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.0502e-05 2021-03-10T10:21:52.030 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "initiated", , 2021-03-10T09:06:01.830 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:07.942287+0000", 2021-03-10T10:21:52.030 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "done", , 2021-03-10T09:06:01.849 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:08.497210+0000", 2021-03-10T10:21:52.031 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "initiated_at": "2021-03-10T09:04:28.375241+0000", , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "age": 93.305483183999996, , 2021-03-10T09:06:01.868 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 4.5438502850000004, 2021-03-10T10:21:52.031 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.887 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:52.032 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:01.906 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "started", 2021-03-10T10:21:52.032 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "flag_point": "started", , 2021-03-10T09:06:01.925 INFO:teuthology.orchestra.run.gibba022.stdout: "client_info": { , 2021-03-10T09:06:01.926 INFO:teuthology.orchestra.run.gibba022.stdout: "client": "client.4370", 2021-03-10T10:21:52.033 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.142+0000 7f31ba7df700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 46087) UID: 0 , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout:{ , 2021-03-10T09:06:02.193 INFO:teuthology.orchestra.run.gibba022.stdout: "ops": [], 2021-03-10T10:21:52.034 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.357 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.358 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "header_read", 2021-03-10T10:21:52.034 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.377 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.00015792399999999999 , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.378 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:52.035 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:16.978389+0000", , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 3.4764999999999997e-05 , 2021-03-10T09:06:02.397 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:52.035 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:17.511406+0000", , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.49168687900000002 , 2021-03-10T09:06:02.417 INFO:teuthology.orchestra.run.gibba022.stdout: } 2021-03-10T10:21:52.036 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:18.215030+0000", , 2021-03-10T09:06:02.437 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.7939000000000001e-05 2021-03-10T10:21:52.037 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", , 2021-03-10T09:06:02.457 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:03:49.483593+0000", 2021-03-10T10:21:52.037 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: { , 2021-03-10T09:06:02.476 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "all_read", 2021-03-10T10:21:52.038 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 6.1550000000000004e-06 , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: }, , 2021-03-10T09:06:02.496 INFO:teuthology.orchestra.run.gibba022.stdout: { 2021-03-10T10:21:52.038 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 0.36809437499999997 , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: } , 2021-03-10T09:06:02.515 INFO:teuthology.orchestra.run.gibba022.stdout: ] 2021-03-10T10:21:52.039 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:26.779007+0000", , 2021-03-10T09:06:02.534 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 2.0275000000000002e-05 , 2021-03-10T09:06:02.535 INFO:teuthology.orchestra.run.gibba022.stdout: }, 2021-03-10T10:21:52.039 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "event": "throttled", , 2021-03-10T09:06:02.553 INFO:teuthology.orchestra.run.gibba022.stdout: "time": "2021-03-10T09:04:31.952403+0000", , 2021-03-10T09:06:02.554 INFO:teuthology.orchestra.run.gibba022.stdout: "duration": 1.9063999999999999e-05 2021-03-10T10:21:52.040 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "queued_for_pg", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:08.896100+0000", , 2021-03-10T09:06:02.952 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.006128568 2021-03-10T10:21:52.040 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "initiated", , 2021-03-10T09:06:02.972 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:03:20.948037+0000", 2021-03-10T10:21:52.041 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "waiting for peered", , 2021-03-10T09:06:02.991 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:26.742027+0000", 2021-03-10T10:21:52.042 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.010 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:52.042 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.029 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "dispatched", 2021-03-10T10:21:52.043 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: { , 2021-03-10T09:06:03.048 INFO:teuthology.orchestra.run.gibba021.stdout: "event": "sub_op_committed", 2021-03-10T10:21:52.043 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.067 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.068 INFO:teuthology.orchestra.run.gibba021.stdout: } 2021-03-10T10:21:52.044 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.086 INFO:teuthology.orchestra.run.gibba021.stdout: "events": [ , 2021-03-10T09:06:03.087 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:52.044 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.105 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 3.3759999999999999e-06 , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.106 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:52.045 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.124 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.8424000000000001e-05 , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.125 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:52.045 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.143 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 0.0018942270000000001 , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.144 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:52.046 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.162 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 6.8745000000000007e-05 , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: }, , 2021-03-10T09:06:03.163 INFO:teuthology.orchestra.run.gibba021.stdout: { 2021-03-10T10:21:52.047 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.181 INFO:teuthology.orchestra.run.gibba021.stdout: "type_data": { , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-10T09:06:03.182 INFO:teuthology.orchestra.run.gibba021.stdout: "client_info": { 2021-03-10T10:21:52.047 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068876+0000", , 2021-03-10T09:06:03.200 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.1307000000000001e-05 , 2021-03-10T09:06:03.201 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:52.048 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "time": "2021-03-10T09:04:31.068948+0000", , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: "duration": 2.666e-06 , 2021-03-10T09:06:03.221 INFO:teuthology.orchestra.run.gibba021.stdout: }, 2021-03-10T10:21:52.048 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: } , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout: ] , 2021-03-10T09:06:03.240 INFO:teuthology.orchestra.run.gibba021.stdout:} 2021-03-10T10:21:52.049 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:Added osd 6 , 2021-03-10T09:06:14.805 INFO:tasks.thrashosds.thrasher:inning osd , 2021-03-10T09:06:14.806 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-03-10T10:21:52.052 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:38.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542220 got 60129542220 for osd.4 , 2021-03-10T09:06:38.912 DEBUG:teuthology.orchestra.run.gibba021:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2021-03-10T09:06:39.136 INFO:teuthology.orchestra.run.gibba021.stdout:674309865515 2021-03-10T10:21:52.053 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2021-03-10T09:06:35.787066+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=35, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.319 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:06:45.320 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2021-03-10T09:06:15.448747+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=15, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:52.055 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2021-03-10T09:06:31.414269+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=31, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.606 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2021-03-10T09:06:37.166077+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=37, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) , 2021-03-10T09:07:05.607 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2021-03-10T09:03:36.130591+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=3, tm_sec=36, tm_wday=2, tm_yday=69, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=10, tm_hour=9, tm_min=6, tm_sec=41, tm_wday=2, tm_yday=69, tm_isdst=0) 2021-03-10T10:21:52.056 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:07:47.243 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2021-03-10T09:07:47.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba021.front.sepia.ceph.com , 2021-03-10T09:07:47.326 DEBUG:teuthology.orchestra.run.gibba021:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-03-10T10:21:52.056 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:=============================================================================== , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 6 +1646us[+1609us] +/- 89ms , 2021-03-10T09:08:56.902 INFO:teuthology.orchestra.run.gibba022.stdout:^* hv02.front.sepia.ceph.com 2 6 377 6 -485us[ -522us] +/- 29ms 2021-03-10T10:21:52.520 INFO:scrape:Found 4 distinct failure reasons 2021-03-10T10:21:52.521 INFO:scrape:Failure: Command failed (workunit test cephadm/create_iscsi_disks.sh) on gibba007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6bef90e2f561208e1aebe76e03aaeb6fae2552e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' 2021-03-10T10:21:52.521 INFO:scrape:4 jobs: ['5952996', '5953002', '5952986', '5952992'] 2021-03-10T10:21:52.522 INFO:scrape:suites intersection: ['task/test_e2e}'] 2021-03-10T10:21:52.522 INFO:scrape:suites union: ['rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable', 'rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing', 'task/test_e2e}'] 2021-03-10T10:21:52.523 INFO:scrape: 2021-03-10T10:21:52.523 INFO:scrape:Failure: Command failed on gibba014 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6046c6c8-817e-11eb-9080-001a4aab830c -e sha1=c6bef90e2f561208e1aebe76e03aaeb6fae2552e -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2021-03-10T10:21:52.523 INFO:scrape:2 jobs: ['5952990', '5953001'] 2021-03-10T10:21:52.524 INFO:scrape:suites intersection: ['1-tasks', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-03-10T10:21:52.524 INFO:scrape:suites union: ['1-tasks', 'distro1$/{ubuntu_18.04}', 'distro1$/{ubuntu_20.04_kubic_stable}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-03-10T10:21:52.525 INFO:scrape: 2021-03-10T10:21:52.525 INFO:scrape:Failure: Command failed on gibba008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:c6bef90e2f561208e1aebe76e03aaeb6fae2552e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3dda861a-8179-11eb-9080-001a4aab830c -- ceph orch host ls --format=json' 2021-03-10T10:21:52.526 INFO:scrape:1 jobs: ['5952983'] 2021-03-10T10:21:52.526 INFO:scrape:suites: ['fixed-2', 'mon_election/classic', 'rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable', 'start}'] 2021-03-10T10:21:52.527 INFO:scrape: 2021-03-10T10:21:52.527 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2021-03-10T10:21:52.527 INFO:scrape:2 jobs: ['5952993', '5952985'] 2021-03-10T10:21:52.528 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'mon_election/connectivity', 'openstack}', 'rados', 'thrashosds-health'] 2021-03-10T10:21:52.528 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}'] 2021-03-10T10:21:52.529 INFO:scrape: 2021-03-10T10:21:52.530 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/ failed: 9 dead: 0 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [5952983] rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:06:57 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952983/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952983/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c209c5e0a8a34684b094ec716ad29da8 Command failed on gibba008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:c6bef90e2f561208e1aebe76e03aaeb6fae2552e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3dda861a-8179-11eb-9080-001a4aab830c -- ceph orch host ls --format=json' [5952985] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:22:36 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952985/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952985/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5952986] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing task/test_e2e} ----------------------------------------------------------------- time: 00:11:32 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952986/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952986/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6819fc03661c451cb713750abaca030c Command failed (workunit test cephadm/create_iscsi_disks.sh) on gibba041 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6bef90e2f561208e1aebe76e03aaeb6fae2552e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5952990] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_20.04_kubic_stable} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:15:29 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952990/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff28bb9c7a0449b4bdfccc3aa126c0ca Command failed on gibba014 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6046c6c8-817e-11eb-9080-001a4aab830c -e sha1=c6bef90e2f561208e1aebe76e03aaeb6fae2552e -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [5952992] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable task/test_e2e} ----------------------------------------------------------------- time: 00:11:15 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952992/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952992/ sentry: https://sentry.ceph.com/organizations/ceph/?query=37ef2fff976c419cbcf641609733f0f0 Command failed (workunit test cephadm/create_iscsi_disks.sh) on gibba040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6bef90e2f561208e1aebe76e03aaeb6fae2552e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5952993] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:13:55 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952993/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952993/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' [5952996] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_testing task/test_e2e} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952996/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952996/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1707519eee1e4adaa4300d2ab79c932b Command failed (workunit test cephadm/create_iscsi_disks.sh) on gibba007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6bef90e2f561208e1aebe76e03aaeb6fae2552e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' [5953001] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_18.04} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:14:11 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953001/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953001/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bcdc3d73da534b4cbb22a5fe161d6079 Command failed on gibba016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b702744-8182-11eb-9080-001a4aab830c -e sha1=c6bef90e2f561208e1aebe76e03aaeb6fae2552e -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [5953002] rados/cephadm/dashboard/{0-distro/ubuntu_20.04_kubic_stable task/test_e2e} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953002/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3dc16cd6b334d5da271b5a0d38cd05e Command failed (workunit test cephadm/create_iscsi_disks.sh) on gibba008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6bef90e2f561208e1aebe76e03aaeb6fae2552e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/cephadm/create_iscsi_disks.sh' Pass ================================================================= [5952982] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:06 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952982/ [5952984] rados/cephadm/thrash/{0-distro/ubuntu_20.04_kubic_testing 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:28:51 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952984/ [5952987] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects} time: 00:21:23 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952987/ [5952988] rados/cephadm/with-work/{0-distro/ubuntu_20.04_kubic_testing fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:33:09 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952988/ [5952989] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:31 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952989/ [5952991] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/lockdep} time: 00:15:16 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952991/ [5952994] rados/cephadm/smoke/{distro/ubuntu_20.04_kubic_stable fixed-2 mon_election/classic start} time: 00:12:59 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952994/ [5952995] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:31:24 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952995/ [5952997] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/radosbench} time: 01:09:12 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952997/ [5952998] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:29:54 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952998/ [5952999] rados/cephadm/thrash/{0-distro/ubuntu_20.04_kubic_testing 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:28:28 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5952999/ [5953000] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:39 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953000/ [5953003] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:51 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953003/ [5953004] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_api_tests} time: 00:27:03 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953004/ [5953005] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:17:37 info: http://pulpito.ceph.com/kchai-2021-03-10_06:55:19-rados-wip-kefu-testing-2021-03-09-1847-distro-basic-gibba/5953005/