2021-09-28T18:52:15.386 INFO:root:teuthology version: 1.1.0-f08b273e 2021-09-28T18:52:15.387 INFO:teuthology.results:subset: '1/10' 2021-09-28T18:52:15.388 INFO:teuthology.results:seed: '8961' 2021-09-29T04:38:50.895 INFO:root:teuthology version: 1.1.0-f08b273e 2021-09-29T04:38:50.958 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-29T05:58:01.528 INFO:teuthology.results:Tests finished! gathering results... 2021-09-29T05:58:22.906 INFO:scrape:Found 200 jobs 2021-09-29T05:58:23.403 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7800-gd35920da (d35920da5e58fb6c116d4e46aa63bce1adc91bff) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7ff21ec3aed9] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55f0eed5ee42] 2021-09-29T05:58:23.404 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680237+0000", , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1848000000000001e-05 , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.405 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680260+0000", , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.429e-06 , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.406 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.025111+0000", , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3769999999999998e-06 , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.406 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030727+0000", , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2734e-05 , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030738+0000", , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.948 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10688 4.1 4:867d4b11:::smithi14022520-2371:head [setxattr _header (58) in=7b,truncate 3832590] snapc 0=[] ondisk+write+known_if_redirected e49)", , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.215697+0000", 2021-09-29T05:58:23.410 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1943000000000001e-05 , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10926 4.1 4:acd8731b:::smithi14022520-2427:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.149252+0000", 2021-09-29T05:58:23.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 10937 , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2021-09-29T05:58:23.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1130000000000001e-05 , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10941 4.1 4:b7809796:::smithi14022520-2431:head [setxattr _header (58) in=7b,truncate 2841010] snapc 0=[] ondisk+write+known_if_redirected e50)", , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.209946+0000", 2021-09-29T05:58:23.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018239060000000001 , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.030763815 , 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058155139999999999 , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.418 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15243790500000001 , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5540999999999999e-05 , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0312e-05 , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15760 4.1 4:a1a32600:::smithi14022520-3488:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.845861+0000", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.802656853000002, 2021-09-29T05:58:23.421 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.699 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.421 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2021-09-29T05:58:23.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15796 4.1 4:b6322122:::smithi14022520-3497:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.999225+0000", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.649292408000001, 2021-09-29T05:58:23.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 26.029215952000001 , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4939999999999997e-06 , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.106965+0000", , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:28.135361+0000", , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6534999999999999e-05 , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.928 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15830 , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.139683+0000", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.314e-05 2021-09-29T05:58:23.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.162774+0000", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7456999999999999e-05 2021-09-29T05:58:23.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.002 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4659", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15834 2021-09-29T05:58:23.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.155334+0000", 2021-09-29T05:58:23.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.047 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.163185+0000", 2021-09-29T05:58:23.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 , 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2021-09-28T19:16:38.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2021-09-29T05:58:23.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:35.951480+0000", 2021-09-29T05:58:23.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:12.550883+0000", , 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 85.894248486999999, , 2021-09-28T19:16:38.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.097021547999999999, 2021-09-29T05:58:23.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2021-09-29T05:58:23.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDECSubOpWrite(3.3s3 49/23 ECSubWrite(tid=2989, reqid=osd.0.13:1382, at_version=0'0, trim_to=0'0, roll_forward_to=49'94))", , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.022112+0000", 2021-09-29T05:58:23.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.903773+0000", , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2638000000000005e-05 , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.984888+0000", , 2021-09-28T19:16:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2021-09-29T05:58:23.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:50.480652+0000", 2021-09-29T05:58:23.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2021-09-29T05:58:23.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0079999999999999e-06 , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025375400000000002 , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:33.359426+0000", , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:34.164662+0000", , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54438732199999995 , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:36.196023+0000", , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9946999999999998e-05 , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:08.809008+0000", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18367588600000001 2021-09-29T05:58:23.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.250 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:13.918374+0000", 2021-09-29T05:58:23.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:15.285507+0000", 2021-09-29T05:58:23.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2021-09-29T05:58:23.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2021-09-28T19:16:39.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1104999999999999e-05 , 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:52.064786+0000", , 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1654000000000001e-05 , 2021-09-28T19:16:39.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.128610159, , 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:39.398 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2021-09-29T05:58:23.449 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", , 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:32.990274+0000", , 2021-09-28T19:16:39.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045450300000000003 2021-09-29T05:58:23.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:00.337 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed , 2021-09-28T19:17:00.457+0000 7fc16ff4d700 -1 osd.1 pg_epoch: 88 pg[2.10( empty local-lis/les=76/77 n=0 ec=51/15 lis/c=76/76 les/c/f=77/77/0 sis=76) [1,0,4] r=0 lpr=76 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x55c50e469600) , 2021-09-28T19:17:02.647+0000 7f7cfe913700 -1 osd.7 pg_epoch: 88 pg[2.0( empty local-lis/les=51/52 n=0 ec=15/15 lis/c=51/51 les/c/f=52/52/0 sis=51) [7,1,0] r=0 lpr=51 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x55914682eb00) 2021-09-29T05:58:23.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:28.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574882 got 55834574882 for osd.5 , 2021-09-28T19:17:28.273 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 , 2021-09-28T19:17:28.547 INFO:teuthology.orchestra.run.smithi140.stdout:55834574887 2021-09-29T05:58:23.456 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:36.317 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2021-09-28T19:17:23.200110+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=23, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2021-09-28T19:17:17.954555+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=17, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2021-09-28T19:16:56.226402+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=16, tm_sec=56, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) 2021-09-29T05:58:23.457 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-09-29T05:58:23.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:19:07.467 INFO:tasks.ceph:Cleaning ceph cluster... , 2021-09-28T19:19:07.468 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2021-09-28T19:19:07.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-09-29T05:58:23.458 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:21:04.842 DEBUG:teuthology.misc:Transferring archived files from smithi140:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411691/remote/smithi140 , 2021-09-28T19:21:04.843 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2021-09-28T19:21:12.723 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2021-09-29T05:58:23.629 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7800-gd35920da (d35920da5e58fb6c116d4e46aa63bce1adc91bff) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7ff21ec3aed9] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55f0eed5ee42] 2021-09-29T05:58:23.631 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680237+0000", , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1848000000000001e-05 , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.631 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680260+0000", , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.429e-06 , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.632 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.025111+0000", , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3769999999999998e-06 , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.633 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030727+0000", , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2734e-05 , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.634 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030738+0000", , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.634 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.635 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.636 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.948 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10688 4.1 4:867d4b11:::smithi14022520-2371:head [setxattr _header (58) in=7b,truncate 3832590] snapc 0=[] ondisk+write+known_if_redirected e49)", , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.215697+0000", 2021-09-29T05:58:23.636 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1943000000000001e-05 , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.637 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.638 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.638 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10926 4.1 4:acd8731b:::smithi14022520-2427:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.149252+0000", 2021-09-29T05:58:23.639 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 10937 , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2021-09-29T05:58:23.640 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1130000000000001e-05 , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.641 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10941 4.1 4:b7809796:::smithi14022520-2431:head [setxattr _header (58) in=7b,truncate 2841010] snapc 0=[] ondisk+write+known_if_redirected e50)", , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.209946+0000", 2021-09-29T05:58:23.641 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018239060000000001 , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.642 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.030763815 , 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.643 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058155139999999999 , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.643 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15243790500000001 , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.644 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5540999999999999e-05 , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.645 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0312e-05 , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.645 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15760 4.1 4:a1a32600:::smithi14022520-3488:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.845861+0000", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.802656853000002, 2021-09-29T05:58:23.646 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.699 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.647 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2021-09-29T05:58:23.647 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.648 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15796 4.1 4:b6322122:::smithi14022520-3497:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.999225+0000", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.649292408000001, 2021-09-29T05:58:23.649 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.649 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 26.029215952000001 , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.650 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4939999999999997e-06 , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.651 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.106965+0000", , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.651 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:28.135361+0000", , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6534999999999999e-05 , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.652 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.928 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15830 , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.653 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.139683+0000", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.314e-05 2021-09-29T05:58:23.653 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.162774+0000", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7456999999999999e-05 2021-09-29T05:58:23.654 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.002 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4659", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15834 2021-09-29T05:58:23.655 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.155334+0000", 2021-09-29T05:58:23.655 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.047 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.163185+0000", 2021-09-29T05:58:23.656 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 , 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2021-09-28T19:16:38.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2021-09-29T05:58:23.657 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:35.951480+0000", 2021-09-29T05:58:23.657 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:12.550883+0000", , 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 85.894248486999999, , 2021-09-28T19:16:38.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.097021547999999999, 2021-09-29T05:58:23.658 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2021-09-29T05:58:23.659 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.660 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDECSubOpWrite(3.3s3 49/23 ECSubWrite(tid=2989, reqid=osd.0.13:1382, at_version=0'0, trim_to=0'0, roll_forward_to=49'94))", , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.022112+0000", 2021-09-29T05:58:23.660 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.903773+0000", , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2638000000000005e-05 , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.661 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.984888+0000", , 2021-09-28T19:16:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2021-09-29T05:58:23.662 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.663 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:50.480652+0000", 2021-09-29T05:58:23.663 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2021-09-29T05:58:23.664 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0079999999999999e-06 , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025375400000000002 , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.666 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:33.359426+0000", , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.667 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:34.164662+0000", , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54438732199999995 , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.667 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:36.196023+0000", , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9946999999999998e-05 , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.668 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:08.809008+0000", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18367588600000001 2021-09-29T05:58:23.669 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.250 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:13.918374+0000", 2021-09-29T05:58:23.669 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:15.285507+0000", 2021-09-29T05:58:23.670 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2021-09-29T05:58:23.671 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2021-09-28T19:16:39.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.671 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1104999999999999e-05 , 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.672 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:52.064786+0000", , 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1654000000000001e-05 , 2021-09-28T19:16:39.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.673 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.128610159, , 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:39.398 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2021-09-29T05:58:23.673 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", , 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:32.990274+0000", , 2021-09-28T19:16:39.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045450300000000003 2021-09-29T05:58:23.674 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:00.337 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed , 2021-09-28T19:17:00.457+0000 7fc16ff4d700 -1 osd.1 pg_epoch: 88 pg[2.10( empty local-lis/les=76/77 n=0 ec=51/15 lis/c=76/76 les/c/f=77/77/0 sis=76) [1,0,4] r=0 lpr=76 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x55c50e469600) , 2021-09-28T19:17:02.647+0000 7f7cfe913700 -1 osd.7 pg_epoch: 88 pg[2.0( empty local-lis/les=51/52 n=0 ec=15/15 lis/c=51/51 les/c/f=52/52/0 sis=51) [7,1,0] r=0 lpr=51 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x55914682eb00) 2021-09-29T05:58:23.677 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:28.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574882 got 55834574882 for osd.5 , 2021-09-28T19:17:28.273 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 , 2021-09-28T19:17:28.547 INFO:teuthology.orchestra.run.smithi140.stdout:55834574887 2021-09-29T05:58:23.679 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:36.317 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2021-09-28T19:17:23.200110+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=23, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2021-09-28T19:17:17.954555+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=17, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2021-09-28T19:16:56.226402+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=16, tm_sec=56, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) 2021-09-29T05:58:23.680 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-09-29T05:58:23.681 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:19:07.467 INFO:tasks.ceph:Cleaning ceph cluster... , 2021-09-28T19:19:07.468 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2021-09-28T19:19:07.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-09-29T05:58:23.681 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:21:04.842 DEBUG:teuthology.misc:Transferring archived files from smithi140:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411691/remote/smithi140 , 2021-09-28T19:21:04.843 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2021-09-28T19:21:12.723 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2021-09-29T05:58:23.835 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7800-gd35920da (d35920da5e58fb6c116d4e46aa63bce1adc91bff) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7ff21ec3aed9] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55f0eed5ee42] 2021-09-29T05:58:23.836 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680237+0000", , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1848000000000001e-05 , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680260+0000", , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.429e-06 , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.838 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.025111+0000", , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3769999999999998e-06 , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.838 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030727+0000", , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2734e-05 , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030738+0000", , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.840 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.840 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.948 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10688 4.1 4:867d4b11:::smithi14022520-2371:head [setxattr _header (58) in=7b,truncate 3832590] snapc 0=[] ondisk+write+known_if_redirected e49)", , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.215697+0000", 2021-09-29T05:58:23.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1943000000000001e-05 , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:23.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10926 4.1 4:acd8731b:::smithi14022520-2427:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.149252+0000", 2021-09-29T05:58:23.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 10937 , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2021-09-29T05:58:23.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1130000000000001e-05 , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10941 4.1 4:b7809796:::smithi14022520-2431:head [setxattr _header (58) in=7b,truncate 2841010] snapc 0=[] ondisk+write+known_if_redirected e50)", , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.209946+0000", 2021-09-29T05:58:23.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018239060000000001 , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.030763815 , 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058155139999999999 , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15243790500000001 , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5540999999999999e-05 , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0312e-05 , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.850 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15760 4.1 4:a1a32600:::smithi14022520-3488:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.845861+0000", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.802656853000002, 2021-09-29T05:58:23.850 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.699 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2021-09-29T05:58:23.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15796 4.1 4:b6322122:::smithi14022520-3497:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.999225+0000", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.649292408000001, 2021-09-29T05:58:23.853 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.853 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 26.029215952000001 , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4939999999999997e-06 , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:23.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.106965+0000", , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.855 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:28.135361+0000", , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6534999999999999e-05 , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.928 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15830 , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.139683+0000", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.314e-05 2021-09-29T05:58:23.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.162774+0000", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7456999999999999e-05 2021-09-29T05:58:23.858 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.002 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4659", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15834 2021-09-29T05:58:23.858 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.155334+0000", 2021-09-29T05:58:23.859 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.047 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.163185+0000", 2021-09-29T05:58:23.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 , 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2021-09-28T19:16:38.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2021-09-29T05:58:23.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:35.951480+0000", 2021-09-29T05:58:23.861 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:12.550883+0000", , 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 85.894248486999999, , 2021-09-28T19:16:38.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.097021547999999999, 2021-09-29T05:58:23.861 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2021-09-29T05:58:23.862 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDECSubOpWrite(3.3s3 49/23 ECSubWrite(tid=2989, reqid=osd.0.13:1382, at_version=0'0, trim_to=0'0, roll_forward_to=49'94))", , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.022112+0000", 2021-09-29T05:58:23.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.903773+0000", , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2638000000000005e-05 , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.864 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.984888+0000", , 2021-09-28T19:16:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2021-09-29T05:58:23.864 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.865 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:50.480652+0000", 2021-09-29T05:58:23.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2021-09-29T05:58:23.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:23.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0079999999999999e-06 , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025375400000000002 , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.868 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:33.359426+0000", , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.869 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:34.164662+0000", , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54438732199999995 , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.869 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:36.196023+0000", , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9946999999999998e-05 , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.870 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:08.809008+0000", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18367588600000001 2021-09-29T05:58:23.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.250 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:13.918374+0000", 2021-09-29T05:58:23.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:15.285507+0000", 2021-09-29T05:58:23.872 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2021-09-29T05:58:23.872 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2021-09-28T19:16:39.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.873 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1104999999999999e-05 , 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:23.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:52.064786+0000", , 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1654000000000001e-05 , 2021-09-28T19:16:39.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:23.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.128610159, , 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:39.398 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2021-09-29T05:58:23.875 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", , 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:32.990274+0000", , 2021-09-28T19:16:39.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045450300000000003 2021-09-29T05:58:23.876 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:00.337 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed , 2021-09-28T19:17:00.457+0000 7fc16ff4d700 -1 osd.1 pg_epoch: 88 pg[2.10( empty local-lis/les=76/77 n=0 ec=51/15 lis/c=76/76 les/c/f=77/77/0 sis=76) [1,0,4] r=0 lpr=76 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x55c50e469600) , 2021-09-28T19:17:02.647+0000 7f7cfe913700 -1 osd.7 pg_epoch: 88 pg[2.0( empty local-lis/les=51/52 n=0 ec=15/15 lis/c=51/51 les/c/f=52/52/0 sis=51) [7,1,0] r=0 lpr=51 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x55914682eb00) 2021-09-29T05:58:23.879 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:28.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574882 got 55834574882 for osd.5 , 2021-09-28T19:17:28.273 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 , 2021-09-28T19:17:28.547 INFO:teuthology.orchestra.run.smithi140.stdout:55834574887 2021-09-29T05:58:23.880 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:36.317 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2021-09-28T19:17:23.200110+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=23, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2021-09-28T19:17:17.954555+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=17, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2021-09-28T19:16:56.226402+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=16, tm_sec=56, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) 2021-09-29T05:58:23.881 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-09-29T05:58:23.882 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:19:07.467 INFO:tasks.ceph:Cleaning ceph cluster... , 2021-09-28T19:19:07.468 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2021-09-28T19:19:07.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-09-29T05:58:23.883 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:21:04.842 DEBUG:teuthology.misc:Transferring archived files from smithi140:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411691/remote/smithi140 , 2021-09-28T19:21:04.843 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2021-09-28T19:21:12.723 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2021-09-29T05:58:24.041 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-7800-gd35920da (d35920da5e58fb6c116d4e46aa63bce1adc91bff) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7ff21ec3aed9] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55f0eed5ee42] 2021-09-29T05:58:24.043 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680237+0000", , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1848000000000001e-05 , 2021-09-28T19:16:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.043 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:04.680260+0000", , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.429e-06 , 2021-09-28T19:16:36.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.044 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.025111+0000", , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3769999999999998e-06 , 2021-09-28T19:16:36.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.045 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030727+0000", , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2734e-05 , 2021-09-28T19:16:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.045 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:36.030738+0000", , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:36.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.046 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.903 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:24.046 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:36.926 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:24.047 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.948 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10688 4.1 4:867d4b11:::smithi14022520-2371:head [setxattr _header (58) in=7b,truncate 3832590] snapc 0=[] ondisk+write+known_if_redirected e49)", , 2021-09-28T19:16:36.949 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.215697+0000", 2021-09-29T05:58:24.048 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1943000000000001e-05 , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:36.971 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:24.048 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", , 2021-09-28T19:16:36.994 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:24.049 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:37.018 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2021-09-29T05:58:24.050 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10926 4.1 4:acd8731b:::smithi14022520-2427:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", , 2021-09-28T19:16:37.041 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.149252+0000", 2021-09-29T05:58:24.050 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 10937 , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.064 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2021-09-29T05:58:24.051 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1130000000000001e-05 , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:24.051 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:10941 4.1 4:b7809796:::smithi14022520-2431:head [setxattr _header (58) in=7b,truncate 2841010] snapc 0=[] ondisk+write+known_if_redirected e50)", , 2021-09-28T19:16:37.110 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:41.209946+0000", 2021-09-29T05:58:24.052 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018239060000000001 , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.053 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.030763815 , 2021-09-28T19:16:37.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.053 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058155139999999999 , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.054 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15243790500000001 , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.055 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5540999999999999e-05 , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.055 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0312e-05 , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.056 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15760 4.1 4:a1a32600:::smithi14022520-3488:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.845861+0000", , 2021-09-28T19:16:37.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.802656853000002, 2021-09-29T05:58:24.056 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.699 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.700 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:24.057 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:37.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2021-09-29T05:58:24.058 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:24.058 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4659.0:15796 4.1 4:b6322122:::smithi14022520-3497:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e61)", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:16:01.999225+0000", , 2021-09-28T19:16:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.649292408000001, 2021-09-29T05:58:24.059 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.060 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 26.029215952000001 , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.060 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4939999999999997e-06 , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2021-09-29T05:58:24.061 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.106965+0000", , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:37.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.061 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:28.135361+0000", , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6534999999999999e-05 , 2021-09-28T19:16:37.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.062 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.928 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15830 , 2021-09-28T19:16:37.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.063 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.139683+0000", , 2021-09-28T19:16:37.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.314e-05 2021-09-29T05:58:24.063 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.162774+0000", , 2021-09-28T19:16:37.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7456999999999999e-05 2021-09-29T05:58:24.064 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.002 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4659", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.140:0/515847703", , 2021-09-28T19:16:38.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 15834 2021-09-29T05:58:24.065 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:38.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:02.155334+0000", 2021-09-29T05:58:24.065 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.047 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", , 2021-09-28T19:16:38.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:36.163185+0000", 2021-09-29T05:58:24.066 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 , 2021-09-28T19:16:38.187 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2021-09-28T19:16:38.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2021-09-29T05:58:24.067 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2021-09-28T19:16:38.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:35.951480+0000", 2021-09-29T05:58:24.067 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:12.550883+0000", , 2021-09-28T19:16:38.509 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 85.894248486999999, , 2021-09-28T19:16:38.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.097021547999999999, 2021-09-29T05:58:24.068 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2021-09-29T05:58:24.068 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2021-09-28T19:16:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.069 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDECSubOpWrite(3.3s3 49/23 ECSubWrite(tid=2989, reqid=osd.0.13:1382, at_version=0'0, trim_to=0'0, roll_forward_to=49'94))", , 2021-09-28T19:16:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2021-09-28T19:15:40.022112+0000", 2021-09-29T05:58:24.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.903773+0000", , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2638000000000005e-05 , 2021-09-28T19:16:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2021-09-28T19:16:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:49.984888+0000", , 2021-09-28T19:16:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2021-09-29T05:58:24.071 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.071 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2021-09-28T19:16:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:50.480652+0000", 2021-09-29T05:58:24.072 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2021-09-29T05:58:24.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2021-09-28T19:16:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2021-09-28T19:16:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2021-09-29T05:58:24.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0079999999999999e-06 , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025375400000000002 , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:33.359426+0000", , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2021-09-28T19:16:39.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:34.164662+0000", , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54438732199999995 , 2021-09-28T19:16:39.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:14:36.196023+0000", , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9946999999999998e-05 , 2021-09-28T19:16:39.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:08.809008+0000", , 2021-09-28T19:16:39.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18367588600000001 2021-09-29T05:58:24.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.250 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2021-09-28T19:16:39.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:13.918374+0000", 2021-09-29T05:58:24.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2021-09-28T19:16:39.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:15.285507+0000", 2021-09-29T05:58:24.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2021-09-28T19:16:39.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2021-09-29T05:58:24.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-09-28T19:16:39.320 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2021-09-28T19:16:39.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1104999999999999e-05 , 2021-09-28T19:16:39.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2021-09-28T19:16:39.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2021-09-29T05:58:24.080 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:15:52.064786+0000", , 2021-09-28T19:16:39.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1654000000000001e-05 , 2021-09-28T19:16:39.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2021-09-29T05:58:24.081 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.128610159, , 2021-09-28T19:16:39.397 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2021-09-28T19:16:39.398 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2021-09-29T05:58:24.081 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", , 2021-09-28T19:16:39.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2021-09-28T19:16:32.990274+0000", , 2021-09-28T19:16:39.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045450300000000003 2021-09-29T05:58:24.082 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:00.337 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed , 2021-09-28T19:17:00.457+0000 7fc16ff4d700 -1 osd.1 pg_epoch: 88 pg[2.10( empty local-lis/les=76/77 n=0 ec=51/15 lis/c=76/76 les/c/f=77/77/0 sis=76) [1,0,4] r=0 lpr=76 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x55c50e469600) , 2021-09-28T19:17:02.647+0000 7f7cfe913700 -1 osd.7 pg_epoch: 88 pg[2.0( empty local-lis/les=51/52 n=0 ec=15/15 lis/c=51/51 les/c/f=52/52/0 sis=51) [7,1,0] r=0 lpr=51 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 1 (0x55914682eb00) 2021-09-29T05:58:24.085 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:28.272 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574882 got 55834574882 for osd.5 , 2021-09-28T19:17:28.273 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 , 2021-09-28T19:17:28.547 INFO:teuthology.orchestra.run.smithi140.stdout:55834574887 2021-09-29T05:58:24.087 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:17:36.317 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2021-09-28T19:17:23.200110+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=23, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2021-09-28T19:17:17.954555+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=17, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) , 2021-09-28T19:17:36.318 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2021-09-28T19:16:56.226402+0000 time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=16, tm_sec=56, tm_wday=1, tm_yday=271, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=9, tm_mday=28, tm_hour=19, tm_min=17, tm_sec=31, tm_wday=1, tm_yday=271, tm_isdst=0) 2021-09-29T05:58:24.087 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-09-29T05:58:24.088 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:19:07.467 INFO:tasks.ceph:Cleaning ceph cluster... , 2021-09-28T19:19:07.468 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2021-09-28T19:19:07.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-09-29T05:58:24.089 WARNING:scrape:Ignoring malparsed backtrace: 2021-09-28T19:21:04.842 DEBUG:teuthology.misc:Transferring archived files from smithi140:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411691/remote/smithi140 , 2021-09-28T19:21:04.843 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2021-09-28T19:21:12.723 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2021-09-29T05:58:24.233 INFO:scrape:Found 2 distinct failure reasons 2021-09-29T05:58:24.234 INFO:scrape:Failure: {'Failure object was': {'smithi007.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} 2021-09-29T05:58:24.235 INFO:scrape:2 jobs: ['6411665', '6411848'] 2021-09-29T05:58:24.235 INFO:scrape:suites intersection: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashosds-health'] 2021-09-29T05:58:24.236 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashers/pggrow', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/cache-pool-snaps}'] 2021-09-29T05:58:24.236 INFO:scrape: 2021-09-29T05:58:24.237 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-09-29T05:58:24.237 INFO:scrape:1 jobs: ['6411691'] 2021-09-29T05:58:24.238 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/cache-agent-big}'] 2021-09-29T05:58:24.238 INFO:scrape: 2021-09-29T05:58:24.246 INFO:teuthology.results:Sending results to to.my.trociny@gmail.com: Test Run: trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/ failed: 1 dead: 2 running: 0 waiting: 0 queued: 0 passed: 197 Fail ================================================================= [6411691] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:15:57 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411691/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411691/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' Dead ================================================================= [6411665] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:04:30 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411665/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411665/ sentry: https://sentry.ceph.com/organizations/ceph/?query=747b9fcb2b2049808a1cbd04cf6be2f0 {'Failure object was': {'smithi007.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75 318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75 318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} [6411848] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:03:49 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411848/ log: http://qa-proxy.ceph.com/teuthology/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411848/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f0109c642834441810b4b63ef071ba5 {'Failure object was': {'smithi057.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75 318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75 318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} Pass ================================================================= [6411642] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:26:00 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411642/ [6411643] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:08:41 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411643/ [6411644] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/redirect} time: 00:17:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411644/ [6411645] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:19:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411645/ [6411646] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:19:21 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411646/ [6411647] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:15:06 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411647/ [6411648] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:22:49 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411648/ [6411649] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:20:59 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411649/ [6411650] rados/dashboard/{centos_8.2_container_tools_3.0 debug/mgr mon_election/classic random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:11:58 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411650/ [6411651] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/sync-many workloads/pool-create-delete} time: 00:18:57 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411651/ [6411652] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:09:37 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411652/ [6411653] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:30 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411653/ [6411654] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:11:31 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411654/ [6411655] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:29:39 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411655/ [6411656] rados/standalone/{supported-random-distro$/{centos_8.stream} workloads/c2c} time: 00:17:37 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411656/ [6411657] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:33:47 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411657/ [6411658] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:39:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411658/ [6411659] rados/upgrade/parallel/{0-distro$/{centos_8.3_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:20:55 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411659/ [6411660] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:28:22 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411660/ [6411661] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:30:24 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411661/ [6411662] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:25 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411662/ [6411663] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:41:10 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411663/ [6411664] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:03 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411664/ [6411666] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:35:37 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411666/ [6411667] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:18:00 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411667/ [6411668] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:35:05 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411668/ [6411669] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:29:42 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411669/ [6411670] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:23:27 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411670/ [6411671] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:38:34 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411671/ [6411672] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache} time: 00:20:16 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411672/ [6411673] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-mixed} time: 00:24:55 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411673/ [6411674] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} time: 00:13:13 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411674/ [6411675] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:31:01 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411675/ [6411676] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:36:26 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411676/ [6411677] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:34:36 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411677/ [6411678] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench} time: 01:34:58 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411678/ [6411679] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect} time: 00:28:58 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411679/ [6411680] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:31:22 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411680/ [6411681] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:16:44 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411681/ [6411682] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:26:00 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411682/ [6411683] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:35:03 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411683/ [6411684] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:26:17 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411684/ [6411685] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects} time: 00:35:19 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411685/ [6411686] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:56 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411686/ [6411687] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:32:48 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411687/ [6411688] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:26:11 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411688/ [6411689] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:20:22 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411689/ [6411690] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:29:46 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411690/ [6411692] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/cache-agent-small} time: 00:21:44 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411692/ [6411693] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:27:50 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411693/ [6411694] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps} time: 00:21:03 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411694/ [6411695] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:26:23 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411695/ [6411696] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:23:44 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411696/ [6411698] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache} time: 00:25:16 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411698/ [6411700] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed} time: 00:13:39 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411700/ [6411701] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} time: 00:14:11 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411701/ [6411703] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/pool-snaps-few-objects} time: 00:27:00 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411703/ [6411705] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/rados_api_tests} time: 00:29:00 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411705/ [6411707] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} time: 00:40:11 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411707/ [6411709] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/radosbench} time: 01:28:28 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411709/ [6411711] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:17:28 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411711/ [6411713] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:11:23 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411713/ [6411714] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:14:58 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411714/ [6411717] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:14:31 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411717/ [6411720] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:23:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411720/ [6411722] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:34:34 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411722/ [6411724] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects} time: 00:22:41 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411724/ [6411726] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:25:28 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411726/ [6411728] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:06 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411728/ [6411730] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:24:11 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411730/ [6411732] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:28:10 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411732/ [6411734] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:35:39 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411734/ [6411736] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-big} time: 00:59:54 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411736/ [6411738] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:25:19 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411738/ [6411740] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:19:58 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411740/ [6411742] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:23:05 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411742/ [6411744] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:37:15 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411744/ [6411746] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:32:20 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411746/ [6411748] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/cache} time: 00:12:24 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411748/ [6411750] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed} time: 00:13:47 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411750/ [6411751] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/dedup-io-snaps} time: 00:12:35 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411751/ [6411753] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:39:51 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411753/ [6411755] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/rados_api_tests} time: 00:26:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411755/ [6411757] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:51:14 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411757/ [6411759] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/radosbench} time: 00:28:58 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411759/ [6411761] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect} time: 00:23:39 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411761/ [6411763] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/redirect_promote_tests} time: 00:13:05 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411763/ [6411765] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect_set_object} time: 00:23:56 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411765/ [6411767] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/set-chunks-read} time: 00:19:56 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411767/ [6411769] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:27:44 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411769/ [6411771] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects-localized} time: 00:23:56 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411771/ [6411773] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/small-objects} time: 00:36:17 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411773/ [6411775] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced} time: 00:36:25 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411775/ [6411777] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:06 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411777/ [6411781] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:38:49 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411781/ [6411785] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:26:28 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411785/ [6411787] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} time: 00:15:21 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411787/ [6411789] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-big} time: 00:23:36 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411789/ [6411791] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-agent-small} time: 00:10:47 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411791/ [6411794] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:27:15 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411794/ [6411797] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps} time: 00:29:57 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411797/ [6411799] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps-balanced} time: 00:22:00 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411799/ [6411801] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:20:56 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411801/ [6411803] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache} time: 00:13:02 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411803/ [6411805] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/dedup-io-mixed} time: 00:09:41 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411805/ [6411807] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/dedup-io-snaps} time: 00:14:46 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411807/ [6411809] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:39:33 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411809/ [6411811] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rados_api_tests} time: 00:44:51 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411811/ [6411813] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} time: 00:22:40 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411813/ [6411815] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:08:16 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411815/ [6411817] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/redirect} time: 00:13:36 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411817/ [6411819] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:23:31 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411819/ [6411821] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:17:57 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411821/ [6411823] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/set-chunks-read} time: 00:19:05 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411823/ [6411825] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} time: 00:22:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411825/ [6411827] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:22:27 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411827/ [6411829] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects} time: 00:18:35 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411829/ [6411830] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:40 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411830/ [6411833] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} time: 00:40:10 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411833/ [6411836] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:41:39 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411836/ [6411838] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:37 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411838/ [6411840] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:23:12 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411840/ [6411842] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/cache-agent-big} time: 00:17:08 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411842/ [6411844] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/cache-agent-small} time: 00:15:29 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411844/ [6411846] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:31:09 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411846/ [6411850] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} time: 00:26:40 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411850/ [6411852] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/cache-snaps} time: 00:28:54 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411852/ [6411854] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/cache} time: 00:15:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411854/ [6411856] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/dedup-io-mixed} time: 00:22:22 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411856/ [6411858] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/dedup-io-snaps} time: 00:25:10 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411858/ [6411860] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:27:21 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411860/ [6411862] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 01:05:41 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411862/ [6411864] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 01:08:35 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411864/ [6411865] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench} time: 01:07:54 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411865/ [6411867] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:16:34 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411867/ [6411869] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:30:44 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411869/ [6411872] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:14:43 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411872/ [6411874] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:26:12 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411874/ [6411876] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:24:14 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411876/ [6411878] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:21:44 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411878/ [6411889] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:18:48 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411889/ [6411891] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:40:13 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411891/ [6411893] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:30:54 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411893/ [6411895] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:26:21 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411895/ [6411897] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:13 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411897/ [6411899] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:25:13 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411899/ [6411901] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:41:06 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411901/ [6411903] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:20:17 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411903/ [6411905] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:26:37 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411905/ [6411907] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:21:39 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411907/ [6411909] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:23:31 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411909/ [6411911] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:24:18 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411911/ [6411913] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:14:05 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411913/ [6411915] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/dedup-io-mixed} time: 00:13:52 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411915/ [6411917] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/dedup-io-snaps} time: 00:17:55 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411917/ [6411920] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:39:08 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411920/ [6411923] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:30:16 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411923/ [6411925] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 01:13:35 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411925/ [6411927] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 00:48:47 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411927/ [6411929] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/redirect} time: 00:19:04 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411929/ [6411931] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} time: 01:40:29 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411931/ [6411933] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:13:44 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411933/ [6411935] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:25:55 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411935/ [6411937] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:16:24 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411937/ [6411939] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:20:30 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411939/ [6411941] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:32:32 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411941/ [6411943] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects} time: 00:19:35 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411943/ [6411947] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:51 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411947/ [6411948] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:08 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411948/ [6411949] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:26:32 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411949/ [6411951] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:14 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411951/ [6411952] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:28:05 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411952/ [6411954] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:47:41 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411954/ [6411956] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:23:56 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411956/ [6411958] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:22:10 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411958/ [6411960] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:31:03 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411960/ [6411962] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:21:58 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411962/ [6411964] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:31:55 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411964/ [6411966] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/cache} time: 00:14:07 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411966/ [6411968] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-mixed} time: 00:22:12 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411968/ [6411970] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} time: 00:21:22 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411970/ [6411972] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:34 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411972/ [6411974] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:26:12 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411974/ [6411976] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 01:00:33 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411976/ [6411978] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench} time: 00:31:21 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411978/ [6411981] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/redirect} time: 00:14:50 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411981/ [6411983] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:17:06 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411983/ [6411985] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:14:09 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411985/ [6411987] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:16:26 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411987/ [6411989] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:28:47 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411989/ [6411991] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:32:46 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411991/ [6411993] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects} time: 00:28:03 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411993/ [6411995] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:38:51 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411995/ [6411997] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:30:53 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411997/ [6411999] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:24:24 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6411999/ [6412001] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:30 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6412001/ [6412003] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:25:22 info: http://pulpito.ceph.com/trociny-2021-09-28_18:51:21-rados-wip-mgolub-testing-distro-basic-smithi/6412003/