Found 311 jobs Ignoring malparsed backtrace: ceph version 17.0.0-14480-gb85c65e6 (b85c65e63903c9c9c3e7a140fe41a7c609105ec3) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x55de78440d0f] , 2: ceph-osd(+0x5d4ed5) [0x55de78440ed5] Ignoring malparsed backtrace: 2022-09-04T14:34:28.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:34:28.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.721968+0000", , 2022-09-04T14:34:28.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7339999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:28.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:34:28.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.727881+0000", , 2022-09-04T14:34:28.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9007999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:28.678 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:28.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.679 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:28.701 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:28.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.701 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:28.723 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:28.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:34:28.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749133+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:28.746 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:28.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:34:28.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749509+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:28.769 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:28.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:34:28.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749532+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:28.791 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:28.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:34:28.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.760924+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:28.815 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:34:28.815 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:34:28.815 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:34:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:28.860 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000560196 , 2022-09-04T14:34:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.883 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:28.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4000000000000001e-06 , 2022-09-04T14:34:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.906 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.272e-06 , 2022-09-04T14:34:28.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.929 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8271999999999997e-05 , 2022-09-04T14:34:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:28.951 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9217999999999998e-05 , 2022-09-04T14:34:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:28.974 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:34:29.136 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops , 2022-09-04T14:34:29.196+0000 7f4052883700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38941) UID: 0 , 2022-09-04T14:34:29.262 INFO:teuthology.orchestra.run.smithi155.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:34:29.427 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:29.427 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:34:29.427 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:29.451 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:34:29.452 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:29.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:29.477 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:29.500 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:29.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1722999999999999e-05 , 2022-09-04T14:34:29.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:29.529 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:29.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00095956199999999996 , 2022-09-04T14:34:29.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:29.552 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:29.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001069609 , 2022-09-04T14:34:29.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:29.575 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:29.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0180000000000004e-06 , 2022-09-04T14:34:29.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:29.598 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:29.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6855999999999999e-05 , 2022-09-04T14:34:29.620 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:29.621 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:34:29.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783087+0000", , 2022-09-04T14:34:29.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:34:29.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:29.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783795+0000", , 2022-09-04T14:34:29.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5680000000000005e-06 , 2022-09-04T14:34:29.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:29.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.794554+0000", , 2022-09-04T14:34:29.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00974504 , 2022-09-04T14:34:29.689 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:29.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:34:29.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.787905+0000", , 2022-09-04T14:34:29.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:34:29.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:34:29.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801911+0000", , 2022-09-04T14:34:29.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7510000000000008e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:29.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:29.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.832190+0000", , 2022-09-04T14:34:29.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018481500000000001 Ignoring malparsed backtrace: 2022-09-04T14:34:29.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:29.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833063+0000", , 2022-09-04T14:34:29.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000120468 Ignoring malparsed backtrace: 2022-09-04T14:34:29.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:29.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833802+0000", , 2022-09-04T14:34:29.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102903 Ignoring malparsed backtrace: 2022-09-04T14:34:30.255 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:30.255 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:30.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:30.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:30.279 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:30.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:30.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:30.303 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:30.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:30.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:30.326 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:30.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:34:30.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:30.348 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:30.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:30.371 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:34:30.372 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:30.372 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:30.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4178999999999999e-05 , 2022-09-04T14:34:30.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:30.395 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:30.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037776599999999997 , 2022-09-04T14:34:30.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:30.418 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:30.441 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:34:30.441 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:34:30.442 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:34:30.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816898+0000", , 2022-09-04T14:34:30.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5265000000000002e-05 , 2022-09-04T14:34:30.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:30.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.817698+0000", , 2022-09-04T14:34:30.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9679999999999995e-06 , 2022-09-04T14:34:30.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:30.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0054464500000000002, , 2022-09-04T14:34:30.511 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:34:30.511 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:34:30.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:34:30.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.841091+0000", , 2022-09-04T14:34:30.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.951e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:30.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:30.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.914176+0000", , 2022-09-04T14:34:30.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1678999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:30.583 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:34:30.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:29.682819+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:30.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:30.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:29.683293+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:30.708 INFO:tasks.rados.rados.0.smithi163.stdout:150: delete oid 11 current snap is 9 , 2022-09-04T14:34:30.692+0000 7f8152796700 -1 osd.3 pg_epoch: 54 pg[3.0s0( v 53'44 (0'0,53'44] local-lis/les=33/34 n=7 ec=33/33 lis/c=33/33 les/c/f=34/34/0 sis=33) [3,7,6,1,5,15,4]p3(0) r=0 lpr=33 crt=53'44 lcod 53'43 mlcod 53'43 active+clean+snaptrim trimq=[2~1] ps=[5~1,7~2]] update_snap_map ::new_snaps:: decode snaps failure on 54'45 (48'27) delete 3:dcdd8930:::smithi16339483-22:8 by unknown.0.0:0 2022-09-04T14:34:30.692804+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 , 2022-09-04T14:34:30.693+0000 7f4031c2c700 -1 osd.7 pg_epoch: 53 pg[3.0s1( v 53'44 (0'0,53'44] local-lis/les=33/34 n=7 ec=33/33 lis/c=33/33 les/c/f=34/34/0 sis=33) [3,7,6,1,5,15,4]p3(0) r=1 lpr=33 luod=0'0 crt=53'44 mlcod 53'44 active mbc={} ps=[5~1,7~2]] update_snap_map ::new_snaps:: decode snaps failure on 54'45 (48'27) delete 3:dcdd8930:::smithi16339483-22:8 by unknown.0.0:0 2022-09-04T14:34:30.692804+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 Ignoring malparsed backtrace: 2022-09-04T14:34:31.234 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops , 2022-09-04T14:34:31.330+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 , 2022-09-04T14:34:31.359 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:34:32.175 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops , 2022-09-04T14:34:32.241+0000 7fdfc7e2d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 39182) UID: 0 , 2022-09-04T14:34:32.299 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:34:33.621 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:34:33.622 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:34:33.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops Ignoring malparsed backtrace: 2022-09-04T14:34:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:34:34.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.744165+0000", , 2022-09-04T14:34:34.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.189e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:34.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.866644+0000", , 2022-09-04T14:34:34.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017599799999999999 Ignoring malparsed backtrace: 2022-09-04T14:34:34.365 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:34:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:34.366 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:34:34.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:13.006678+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:34.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:34.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783476+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:34:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:34:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:34:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:34:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:34:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:34:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:34:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.111e-05 , 2022-09-04T14:34:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:34.563 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026451629999999999 , 2022-09-04T14:34:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:34:34.586 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:34:34.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.843630+0000", , 2022-09-04T14:34:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:34:34.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:24.831567+0000", , 2022-09-04T14:34:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0383e-05 , 2022-09-04T14:34:34.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.786616+0000", , 2022-09-04T14:34:34.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089162890000000009 , 2022-09-04T14:34:34.657 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:34.944 INFO:teuthology.orchestra.run.smithi077.stdout:{ , 2022-09-04T14:34:34.945 INFO:teuthology.orchestra.run.smithi077.stdout: "ops": [], , 2022-09-04T14:34:34.945 INFO:teuthology.orchestra.run.smithi077.stdout: "complaint_time": 30, Ignoring malparsed backtrace: 2022-09-04T14:34:35.100 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:35.100 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:35.100 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.781888+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:35.123 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.123 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:35.123 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:34:35.146 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.146 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:35.147 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:35.169 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.169 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:35.169 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:34:35.192 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:34:35.192 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.192 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:35.215 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.0289999999999996e-06 , 2022-09-04T14:34:35.215 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.215 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:35.238 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.2169000000000003e-05 , 2022-09-04T14:34:35.238 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.238 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:35.262 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816324+0000", , 2022-09-04T14:34:35.262 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:34:35.262 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:35.285 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.819551+0000", , 2022-09-04T14:34:35.285 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.0845e-05 , 2022-09-04T14:34:35.285 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:35.307 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833317+0000", , 2022-09-04T14:34:35.307 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.9665000000000001e-05 , 2022-09-04T14:34:35.308 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:35.330 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:34:35.330 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.837366+0000", , 2022-09-04T14:34:35.330 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:35.353 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:34:35.353 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:27.780189+0000", , 2022-09-04T14:34:35.353 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.5279000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:35.375 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:34:35.375 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686278+0000", , 2022-09-04T14:34:35.376 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.6582000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:35.398 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:35.398 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:34:35.398 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.683928+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:35.420 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:35.420 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:34:35.421 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:34.349772+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:35.847 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.035869856999999998 , 2022-09-04T14:34:35.848 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.848 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:35.876 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.6820000000000003e-06 , 2022-09-04T14:34:35.877 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:35.877 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:35.905 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:34:35.905 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:34:35.906 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:34:35.940 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.809456+0000", , 2022-09-04T14:34:35.941 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.6879000000000003e-05 , 2022-09-04T14:34:35.941 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:35.966 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816108+0000", , 2022-09-04T14:34:35.966 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.6193000000000001e-05 , 2022-09-04T14:34:35.967 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:35.996 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0034931129999999999, , 2022-09-04T14:34:35.996 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:34:35.997 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:34:36.036 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:34:36.036 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.817544+0000", , 2022-09-04T14:34:36.036 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.001314953 Ignoring malparsed backtrace: 2022-09-04T14:34:36.061 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:34:36.062 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831851+0000", , 2022-09-04T14:34:36.062 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.2479999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:36.090 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 11.998151689, , 2022-09-04T14:34:36.090 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.01066607, , 2022-09-04T14:34:36.090 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:34:36.114 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:36.115 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:34:36.115 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:24.598343+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:36.143 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:36.143 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:34:36.144 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.842510+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:36.176 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:23.843585+0000", , 2022-09-04T14:34:36.176 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 11.988955996, , 2022-09-04T14:34:36.177 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0058364949999999997, Ignoring malparsed backtrace: 2022-09-04T14:34:36.209 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:36.209 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:36.209 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:36.232 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:36.232 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:36.232 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:34:36.255 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.cs6 58/33 ECSubWrite(tid=29, reqid=osd.14.0:29, at_version=58'23, trim_to=0'0, roll_forward_to=55'21))", , 2022-09-04T14:34:36.255 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:35.022033+0000", , 2022-09-04T14:34:36.256 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 0.81050767999999995, Ignoring malparsed backtrace: 2022-09-04T14:34:36.540 INFO:teuthology.orchestra.run.smithi077.stdout:} , 2022-09-04T14:34:36.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops , 2022-09-04T14:34:36.551 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} Ignoring malparsed backtrace: 2022-09-04T14:34:37.903 INFO:tasks.rados.rados.0.smithi163.stdout:179: finishing write tid 1 to smithi16339483-19 , 2022-09-04T14:34:37.903 INFO:tasks.rados.rados.0.smithi163.stdout:181: seq_num 94 ranges {2408448=425984,2834432=786432,3620864=376832} , noscrub is unset Ignoring malparsed backtrace: 2022-09-04T14:34:38.902+0000 7fc7f1867700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset c=[]:{c=[c,b,9]} for object 3:da2cfe71:::smithi16339483-46:7 , 2022-09-04T14:34:38.902+0000 7f793eff5700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2=[]:{} for object 3:a9d45bed:::smithi16339483-27:2 , 2022-09-04T14:34:38.901+0000 7f8152796700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset e=[]:{e=[e,c,b,9]} for object 3:0dcdd893:::smithi16339483-22:8 Ignoring malparsed backtrace: 2022-09-04T14:34:39.339 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:34:39.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.728035+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:39.367 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.367 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:34:39.368 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.728058+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:39.392 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.392 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", , 2022-09-04T14:34:39.392 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.763618+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:39.414 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.414 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:34:39.415 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778570+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:39.437 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:34:39.437 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:34:39.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:39.460 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:34:39.460 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:34:39.460 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:34:39.482 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:34:39.483 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:34:39.483 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:34:39.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:39.506 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:34:39.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:39.528 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:34:39.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:39.559 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.559 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2022-09-04T14:34:39.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:39.581 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.582 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2022-09-04T14:34:39.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:39.604 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.604 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:34:39.627 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:34:39.627 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:34:39.627 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:39.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3510000000000002e-06 , 2022-09-04T14:34:39.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:39.650 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:39.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079662129999999998 , 2022-09-04T14:34:39.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:39.673 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:39.696 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:39.696 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.0s1 53/33 ECSubWrite(tid=69, reqid=client.4557.0:349, at_version=53'39, trim_to=0'0, roll_forward_to=51'37))", , 2022-09-04T14:34:39.696 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:29.684588+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:39.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.684650+0000", , 2022-09-04T14:34:39.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5908999999999999e-05 , 2022-09-04T14:34:39.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:40.220 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, , 2022-09-04T14:34:40.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, , 2022-09-04T14:34:40.220 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ Ignoring malparsed backtrace: 2022-09-04T14:34:40.246 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:34:40.247 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:34:40.247 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:34:40.271 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:34:40.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:34:40.272 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:34:40.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:40.297 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:40.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:40.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:40.322 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:40.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:34:40.345 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.5s6 47/33 ECSubWrite(tid=19, reqid=client.4557.0:298, at_version=47'13, trim_to=0'0, roll_forward_to=37'11))", , 2022-09-04T14:34:40.346 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.832076+0000", , 2022-09-04T14:34:40.346 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.384706832999999, Ignoring malparsed backtrace: 2022-09-04T14:34:40.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3574000000000001e-05 , 2022-09-04T14:34:40.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:40.369 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:40.392 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8254e-05 , 2022-09-04T14:34:40.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:40.393 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:40.415 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:40.415 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.5s6 47/33 ECSubWrite(tid=20, reqid=client.4557.0:299, at_version=47'14, trim_to=0'0, roll_forward_to=37'11))", , 2022-09-04T14:34:40.416 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.833001+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:40.439 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833761+0000", , 2022-09-04T14:34:40.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5441e-05 , 2022-09-04T14:34:40.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:40.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.834519+0000", , 2022-09-04T14:34:40.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9816999999999997e-05 , 2022-09-04T14:34:40.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:40.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:40.485 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:40.485 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.5s6 47/33 ECSubWrite(tid=22, reqid=client.4557.0:300, at_version=0'0, trim_to=0'0, roll_forward_to=47'15))", Ignoring malparsed backtrace: 2022-09-04T14:34:40.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:34:40.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:24.829161+0000", , 2022-09-04T14:34:40.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4477999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:40.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:40.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:24.838655+0000", , 2022-09-04T14:34:40.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6159e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:40.554 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:34:40.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:40.555 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:40.578 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:40.578 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:34:40.578 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.682750+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:40.950 INFO:tasks.rados.rados.0.smithi163.stdout:201: writing smithi16339483-36 from 1671168 to 2228224 tid 3 , 2022-09-04T14:34:40.950 INFO:tasks.rados.rados.0.smithi163.stdout:201: writing smithi16339483-36 from 2228224 to 2244608 tid 4 , 2022-09-04T14:34:40.951 INFO:tasks.rados.rados.0.smithi163.stdout:194: done (4 left) Ignoring malparsed backtrace: 2022-09-04T14:34:41.106 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:34:41.106 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:34:41.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:41.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:41.130 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:41.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:34:41.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:41.153 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:41.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:34:41.177 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:34:41.177 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:34:41.177 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:41.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 , 2022-09-04T14:34:41.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:41.201 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:41.224 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5520000000000001e-06 , 2022-09-04T14:34:41.224 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:41.224 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:41.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030003320000000001 , 2022-09-04T14:34:41.249 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:34:41.249 INFO:teuthology.orchestra.run.smithi163.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:34:41.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683326+0000", , 2022-09-04T14:34:41.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:34:41.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:41.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.693496+0000", , 2022-09-04T14:34:41.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1265e-05 , 2022-09-04T14:34:41.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:41.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.702103+0000", , 2022-09-04T14:34:41.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044592709999999999 , 2022-09-04T14:34:41.320 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:41.343 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:34:41.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.699513+0000", , 2022-09-04T14:34:41.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:34:41.366 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:34:41.366 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.701381+0000", , 2022-09-04T14:34:41.367 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4180000000000003e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:41.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:34:41.390 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.739476+0000", , 2022-09-04T14:34:41.390 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057067639999999996 Ignoring malparsed backtrace: 2022-09-04T14:34:41.413 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:41.413 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:34:41.414 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.736069+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:41.438 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:41.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:34:41.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:35.023416+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:41.980 INFO:tasks.rados.rados.0.smithi163.stdout:207: finishing write tid 1 to smithi16339483-1 , 2022-09-04T14:34:41.981 INFO:tasks.rados.rados.0.smithi163.stdout:207: finishing write tid 2 to smithi16339483-1 , 2022-09-04T14:34:41.981 INFO:tasks.rados.rados.0.smithi163.stdout:207: finishing write tid 3 to smithi16339483-1 Ignoring malparsed backtrace: 2022-09-04T14:34:42.995 INFO:tasks.rados.rados.0.smithi163.stdout:append oid 31 current snap is 16 , 2022-09-04T14:34:42.996 INFO:tasks.rados.rados.0.smithi163.stdout:217: seq_num 107 ranges {507904=638976,1146880=458752,1605632=704512} , 2022-09-04T14:34:43.001 INFO:tasks.rados.rados.0.smithi163.stdout:217: writing smithi16339483-31 from 507904 to 1146880 tid 1 Ignoring malparsed backtrace: 2022-09-04T14:34:44.545 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} , 2022-09-04T14:34:44.561 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} , 2022-09-04T14:34:44.578+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:34:44.660 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.781114+0000", , 2022-09-04T14:34:44.660 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:34:44.660 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:44.684 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.783312+0000", , 2022-09-04T14:34:44.684 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.1254000000000002e-05 , 2022-09-04T14:34:44.685 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:44.716 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.785248+0000", , 2022-09-04T14:34:44.716 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.0913999999999999e-05 , 2022-09-04T14:34:44.717 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:44.741 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:34:44.742 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.815333+0000", , 2022-09-04T14:34:44.742 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:44.770 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:34:44.770 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.815729+0000", , 2022-09-04T14:34:44.770 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00013038700000000001 Ignoring malparsed backtrace: 2022-09-04T14:34:44.795 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:44.796 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.817034+0000", , 2022-09-04T14:34:44.796 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.4042999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:44.819 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:44.820 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:34:44.820 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831072+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:44.849 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:44.849 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:34:44.849 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833001+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:44.872 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:44.872 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:44.872 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833638+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:44.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:44.894 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:44.895 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:34:44.917 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:44.917 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:44.917 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:44.940 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:44.940 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:34:44.940 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:34:44.963 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:34:44.964 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:44.964 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:44.986 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.511e-06 , 2022-09-04T14:34:44.987 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:44.987 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:45.009 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.2585000000000001e-05 , 2022-09-04T14:34:45.009 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:34:45.009 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:34:45.170 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-13, ['dump_ops_in_flight'] , 2022-09-04T14:34:45.215+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:34:45.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:15.746602+0000", , 2022-09-04T14:34:45.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:45.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:45.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.800080+0000", , 2022-09-04T14:34:45.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0241000000000001e-05 , 2022-09-04T14:34:45.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:45.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.804089+0000", , 2022-09-04T14:34:45.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013259529999999999 , 2022-09-04T14:34:45.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:45.539 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:34:45.539 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 274 , 2022-09-04T14:34:45.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:45.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:34:45.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828486+0000", , 2022-09-04T14:34:45.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:45.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:34:45.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828913+0000", , 2022-09-04T14:34:45.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3759999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:45.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:34:45.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828950+0000", , 2022-09-04T14:34:45.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8399999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:45.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:45.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.819377+0000", , 2022-09-04T14:34:45.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019263799999999999 Ignoring malparsed backtrace: 2022-09-04T14:34:45.670 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:34:45.670 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:34:45.671 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 290 Ignoring malparsed backtrace: 2022-09-04T14:34:45.695 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:45.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:34:45.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.832138+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:45.724 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:45.725 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:45.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:45.749 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:45.749 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:45.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:45.773 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:45.774 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:45.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:45.802 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:45.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:45.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:45.831 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:45.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:45.854 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:45.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:45.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub , 2022-09-04T14:34:45.965+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 , 2022-09-04T14:34:45.965+0000 7fc7f1867700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset c=[]:{c=[c,b,9]} for object 3:da2cfe71:::smithi16339483-46:7 Ignoring malparsed backtrace: 2022-09-04T14:34:46.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:46.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.732880+0000", , 2022-09-04T14:34:46.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016545780000000001 Ignoring malparsed backtrace: 2022-09-04T14:34:46.400 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.630794276000003, , 2022-09-04T14:34:46.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019991169999999999, , 2022-09-04T14:34:46.400 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:34:46.425 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:46.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:34:46.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.722467+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:46.449 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:34:46.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.722488+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:46.478 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:46.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749582+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:46.504 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:46.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:46.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.761458+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:46.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.762628+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:34:46.554 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:46.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:46.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:46.583 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:46.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:46.608 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:34:46.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:34:46.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:34:46.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6290000000000004e-06 , 2022-09-04T14:34:46.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:46.634 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:46.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057493500000000005 , 2022-09-04T14:34:46.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:46.659 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:46.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000418226 , 2022-09-04T14:34:46.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:46.691 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010721400000000001 , 2022-09-04T14:34:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 198 , 2022-09-04T14:34:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:34:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.779834+0000", , 2022-09-04T14:34:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:47.317 INFO:teuthology.orchestra.run.smithi155.stdout:{ , 2022-09-04T14:34:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, , 2022-09-04T14:34:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, Ignoring malparsed backtrace: 2022-09-04T14:34:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:34:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:34:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:34:47.372 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:34:47.372 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:34:47.373 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:34:47.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:34:47.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:47.396 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:47.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:34:47.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:47.420 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:47.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3080000000000001e-05 , 2022-09-04T14:34:47.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:47.443 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:47.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.186883469 , 2022-09-04T14:34:47.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:47.467 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:47.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.18772504500000001 , 2022-09-04T14:34:47.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:47.490 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:47.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3115999999999995e-05 , 2022-09-04T14:34:47.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:47.513 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:47.536 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:34:47.536 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:34:47.536 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:34:47.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783729+0000", , 2022-09-04T14:34:47.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4223000000000001e-05 , 2022-09-04T14:34:47.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:47.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784738+0000", , 2022-09-04T14:34:47.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7540000000000006e-06 , 2022-09-04T14:34:47.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:47.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0068225359999999997, , 2022-09-04T14:34:47.606 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:34:47.606 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:34:47.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:34:47.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801835+0000", , 2022-09-04T14:34:47.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7821999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:47.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.832267+0000", , 2022-09-04T14:34:47.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021792230000000001 Ignoring malparsed backtrace: 2022-09-04T14:34:47.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:47.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833134+0000", , 2022-09-04T14:34:47.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030157069999999998 Ignoring malparsed backtrace: 2022-09-04T14:34:47.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:47.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.908286+0000", , 2022-09-04T14:34:47.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003130673 Ignoring malparsed backtrace: 2022-09-04T14:34:47.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:47.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.909173+0000", , 2022-09-04T14:34:47.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003982925 Ignoring malparsed backtrace: 2022-09-04T14:34:48.145 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} , 2022-09-04T14:34:48.167 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} , 2022-09-04T14:34:48.189 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} Ignoring malparsed backtrace: 2022-09-04T14:34:48.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.760134+0000", , 2022-09-04T14:34:48.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0066417179999999996 , 2022-09-04T14:34:48.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:48.321 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:34:48.322 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 157 , 2022-09-04T14:34:48.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:48.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:34:48.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858431+0000", , 2022-09-04T14:34:48.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:48.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:48.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.800263+0000", , 2022-09-04T14:34:48.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1044999999999995e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:48.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:48.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.804062+0000", , 2022-09-04T14:34:48.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5982e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:48.427 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:48.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:34:48.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.815822+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:48.449 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:48.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:34:48.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816639+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:48.472 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:48.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:48.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.817091+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:48.494 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:48.494 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:48.495 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:48.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:48.518 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:48.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:34:48.541 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:48.541 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:34:48.541 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:48.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:48.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:48.564 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:48.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5970000000000003e-06 , 2022-09-04T14:34:48.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:48.586 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:48.608 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 580 , 2022-09-04T14:34:48.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:48.609 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:34:48.631 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 581 , 2022-09-04T14:34:48.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:48.631 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: osd.0: osd_enable_op_tracker = 'false' , osd.1: osd_enable_op_tracker = 'false' , osd.2: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:34:50.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops , 2022-09-04T14:34:50.371+0000 7faf5177f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38712) UID: 0 , 2022-09-04T14:34:50.466 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:34:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.830998+0000", , 2022-09-04T14:34:50.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5644000000000001e-05 , 2022-09-04T14:34:50.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:34:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.835788+0000", , 2022-09-04T14:34:51.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:34:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.836378+0000", , 2022-09-04T14:34:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046487399999999999 Ignoring malparsed backtrace: 2022-09-04T14:34:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.831773+0000", , 2022-09-04T14:34:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1087999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:34:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.831488+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:34:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834830+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.843032+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:51.156 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:34:51.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:51.179 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:34:51.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:34:51.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:51.203 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:51.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3399999999999997e-06 , 2022-09-04T14:34:51.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:51.226 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:51.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6796999999999999e-05 , 2022-09-04T14:34:51.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:51.250 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:51.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013747400000000001 , 2022-09-04T14:34:51.273 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:34:51.273 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:34:51.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:34:51.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", , 2022-09-04T14:34:51.297 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:34:51.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:46.868607+0000", , 2022-09-04T14:34:51.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4343e-05 , 2022-09-04T14:34:51.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 39.058312667000003, , 2022-09-04T14:34:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026361870999999999, , 2022-09-04T14:34:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:34:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:34:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.723084+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:34:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.725250+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:34:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.737850+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.880 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:34:51.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.750968+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.904 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:34:51.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.753684+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:51.930 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:12.757603+0000", , 2022-09-04T14:34:51.930 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 39.023472267999999, , 2022-09-04T14:34:51.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014385123999999999, Ignoring malparsed backtrace: 2022-09-04T14:34:51.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:51.954 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:34:51.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:51.977 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:51.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:34:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:34:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:34:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:34:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2062000000000001e-05 , 2022-09-04T14:34:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022800191000000001 , 2022-09-04T14:34:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:34:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:34:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916278+0000", , 2022-09-04T14:34:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.999999 , 2022-09-04T14:34:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.963898+0000", , 2022-09-04T14:34:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6514000000000001e-05 , 2022-09-04T14:34:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:52.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:34:52.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077210+0000", , 2022-09-04T14:34:52.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:52.353 INFO:teuthology.orchestra.run.smithi038.stdout:{ , 2022-09-04T14:34:52.354 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], , 2022-09-04T14:34:52.354 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:52.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:34:52.743 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:34:52.743 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:52.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6149000000000001e-05 , 2022-09-04T14:34:52.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:52.766 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:52.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4039999999999998e-06 , 2022-09-04T14:34:52.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:52.789 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:52.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:34:52.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:34:52.812 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:34:52.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.778780+0000", , 2022-09-04T14:34:52.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7736e-05 , 2022-09-04T14:34:52.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:52.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682985+0000", , 2022-09-04T14:34:52.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0839999999999992e-06 , 2022-09-04T14:34:52.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:52.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13640564299999999, , 2022-09-04T14:34:52.882 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:34:52.882 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:34:52.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:34:52.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.560441+0000", , 2022-09-04T14:34:52.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.40320693000000002 Ignoring malparsed backtrace: 2022-09-04T14:34:52.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:34:52.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.908377+0000", , 2022-09-04T14:34:52.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7934000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:52.960 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.806120141999999, , 2022-09-04T14:34:52.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012096347, , 2022-09-04T14:34:52.960 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:34:52.985 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:52.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:34:52.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:41.634468+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:53.011 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", , 2022-09-04T14:34:53.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.966492+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:53.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.036 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:34:53.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.062 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:34:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783381+0000", , 2022-09-04T14:34:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3348000000000001e-05 , 2022-09-04T14:34:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:53.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.789791+0000", , 2022-09-04T14:34:53.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025171820000000002 , 2022-09-04T14:34:53.649 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:34:53.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:34:53.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.832823+0000", , 2022-09-04T14:34:53.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967296 Ignoring malparsed backtrace: 2022-09-04T14:34:53.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:34:53.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833188+0000", , 2022-09-04T14:34:53.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8930000000000005e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:53.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:34:53.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.840927+0000", , 2022-09-04T14:34:53.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067311599999999999 Ignoring malparsed backtrace: 2022-09-04T14:34:53.763 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:34:53.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834326+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:53.789 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:34:53.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.777357+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:53.813 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:34:53.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.786750+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:53.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.837 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:34:53.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.860 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:34:53.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.884 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:34:53.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:34:53.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:34:53.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.908 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:53.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7378000000000004e-05 , 2022-09-04T14:34:53.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.932 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:53.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115835 , 2022-09-04T14:34:53.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:34:53.955 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:53.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.957753+0000", , 2022-09-04T14:34:53.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:34:53.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: osd.10: osd_enable_op_tracker = 'false' , osd.11: osd_enable_op_tracker = 'false' , osd.12: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:34:55.203+0000 7ffa4ea67700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 38946) UID: 0 , 2022-09-04T14:34:55.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight , 2022-09-04T14:34:55.304 INFO:teuthology.orchestra.run.smithi077.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:34:56.589 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.722026+0000", , 2022-09-04T14:34:56.589 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.862588590000001, , 2022-09-04T14:34:56.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0063489840000000002, Ignoring malparsed backtrace: 2022-09-04T14:34:56.614 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:56.614 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:56.639 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:34:56.639 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:34:56.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.673 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:56.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.700 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:34:56.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.723 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:34:56.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.746 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:34:56.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.770 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:56.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.794 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:56.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:34:56.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:56.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.818 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:56.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2069999999999998e-06 , 2022-09-04T14:34:56.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.842 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:56.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9336000000000002e-05 , 2022-09-04T14:34:56.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:56.865 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.787905+0000", , 2022-09-04T14:34:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801842+0000", , 2022-09-04T14:34:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5440000000000001e-06 , 2022-09-04T14:34:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.832369+0000", , 2022-09-04T14:34:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101374 , 2022-09-04T14:34:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833229+0000", , 2022-09-04T14:34:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4796000000000005e-05 , 2022-09-04T14:34:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:56.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.909380+0000", , 2022-09-04T14:34:56.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020620100000000001 , 2022-09-04T14:34:56.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:57.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:47.970778+0000", , 2022-09-04T14:34:57.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6741000000000001e-05 , 2022-09-04T14:34:57.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:57.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6939999999999998e-06 , 2022-09-04T14:34:57.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:57.433 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:57.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8250000000000001e-06 , 2022-09-04T14:34:57.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:57.459 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:57.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14390066900000001 , 2022-09-04T14:34:57.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:57.486 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:57.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0033816559999999998 , 2022-09-04T14:34:57.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:57.516 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:57.546 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801792+0000", , 2022-09-04T14:34:57.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:34:57.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:57.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.815461+0000", , 2022-09-04T14:34:57.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3650000000000004e-06 , 2022-09-04T14:34:57.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:57.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816162+0000", , 2022-09-04T14:34:57.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2377000000000002e-05 , 2022-09-04T14:34:57.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:57.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:34:57.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816842+0000", , 2022-09-04T14:34:57.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:34:57.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:34:57.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833042+0000", , 2022-09-04T14:34:57.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1269999999999992e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:57.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:34:57.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.843567+0000", , 2022-09-04T14:34:57.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6615999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:57.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:34:57.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.914388+0000", , 2022-09-04T14:34:57.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3522e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:57.725 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:57.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:34:57.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:29.683304+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:57.750 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:34:57.750 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:34:57.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:57.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:57.776 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:57.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:34:57.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:57.801 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:57.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:34:57.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:34:57.826 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:34:57.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:34:58.132 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops , 2022-09-04T14:34:58.140+0000 7faf5177f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38712) UID: 0 , 2022-09-04T14:34:58.242+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:34:58.288 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:58.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:34:58.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.779827+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:58.323 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:58.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:34:58.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.779859+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:58.347 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:58.347 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:34:58.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.800358+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:58.370 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:58.371 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:34:58.371 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.801933+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:58.394 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:34:58.394 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:34:58.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:58.417 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:34:58.417 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:34:58.418 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:34:58.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:58.441 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:58.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:34:58.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:58.465 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:58.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:34:58.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:58.488 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:34:58.488 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:58.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:34:58.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:58.512 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:58.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.422e-06 , 2022-09-04T14:34:58.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:58.535 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:58.558 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.579e-06 , 2022-09-04T14:34:58.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:58.559 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:58.582 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4079000000000005e-05 , 2022-09-04T14:34:58.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:58.582 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:58.605 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:58.606 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.4s5 69/52 ECSubWrite(tid=35, reqid=client.4557.0:590, at_version=69'18, trim_to=0'0, roll_forward_to=67'16))", , 2022-09-04T14:34:58.606 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:45.963688+0000", Ignoring malparsed backtrace: 2022-09-04T14:34:58.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.073651+0000", , 2022-09-04T14:34:58.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 , 2022-09-04T14:34:58.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:58.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.073718+0000", , 2022-09-04T14:34:58.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7889999999999997e-06 , 2022-09-04T14:34:58.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:59.077 INFO:tasks.rados.rados.0.smithi163.stdout:287: done (7 left) , 2022-09-04T14:34:59.078 INFO:tasks.rados.rados.0.smithi163.stdout:288: rmattr oid 32 current snap is 23 , 2022-09-04T14:34:59.078 INFO:tasks.rados.rados.0.smithi163.stdout:append oid 35 current snap is 23 Ignoring malparsed backtrace: 2022-09-04T14:34:59.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.203 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:59.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:34:59.227 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:69 3.9s0 3:97bfca98:::smithi16339483-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:34:59.227 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:12.754546+0000", , 2022-09-04T14:34:59.228 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 46.424902176000003, Ignoring malparsed backtrace: 2022-09-04T14:34:59.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:34:59.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.250 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:34:59.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.274 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1900000000000003e-06 , 2022-09-04T14:34:59.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.297 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1576999999999995e-05 , 2022-09-04T14:34:59.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.320 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.342 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2256999999999994e-05 , 2022-09-04T14:34:59.342 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.342 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.364 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010191699999999999 , 2022-09-04T14:34:59.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.365 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.388 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105621 , 2022-09-04T14:34:59.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.388 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.410 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0068205699999999998 , 2022-09-04T14:34:59.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.411 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:34:59.433 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 208 , 2022-09-04T14:34:59.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.433 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:34:59.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.787775+0000", , 2022-09-04T14:34:59.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:34:59.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:34:59.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:34:59.478 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:34:59.479 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.cs4 47/33 ECSubWrite(tid=21, reqid=client.4557.0:312, at_version=47'15, trim_to=0'0, roll_forward_to=45'11))", Ignoring malparsed backtrace: 2022-09-04T14:34:59.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:34:59.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680147+0000", , 2022-09-04T14:34:59.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:34:59.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:34:59.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680197+0000", , 2022-09-04T14:34:59.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3619999999999997e-06 Ignoring malparsed backtrace: 2022-09-04T14:34:59.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:34:59.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.684959+0000", , 2022-09-04T14:34:59.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2116000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:34:59.575 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:34:59.575 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.695931+0000", , 2022-09-04T14:34:59.575 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010524250000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:00.056 INFO:tasks.rados.rados.0.smithi163.stdout:update_object_version oid 29 v 21 (ObjNum 122 snap 23 seq_num 122) dirty exists , 2022-09-04T14:35:00.056 INFO:tasks.rados.rados.0.smithi163.stdout:286: left oid 29 (ObjNum 122 snap 23 seq_num 122) , 2022-09-04T14:35:00.056 INFO:tasks.rados.rados.0.smithi163.stdout:290: finishing write tid 6 to smithi16339483-5 Ignoring malparsed backtrace: 2022-09-04T14:35:00.233 INFO:tasks.rados.rados.0.smithi163.stdout:303: finishing write tid 5 to smithi16339483-19 , 2022-09-04T14:35:00.233 INFO:tasks.rados.rados.0.smithi163.stdout:304: finishing write tid 1 to smithi16339483-31 , 2022-09-04T14:35:00.233 INFO:tasks.rados.rados.0.smithi163.stdout:303: finishing write tid 6 to smithi16339483-19 Ignoring malparsed backtrace: noscrub is unset , 2022-09-04T14:35:02.080 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} , 2022-09-04T14:35:02.106 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} Ignoring malparsed backtrace: 2022-09-04T14:35:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3348000000000001e-05 , 2022-09-04T14:35:02.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:02.463 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025171820000000002 , 2022-09-04T14:35:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:35:02.493 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:35:02.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.832823+0000", , 2022-09-04T14:35:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967296 , 2022-09-04T14:35:02.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:02.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833188+0000", , 2022-09-04T14:35:02.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8930000000000005e-06 , 2022-09-04T14:35:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.840927+0000", , 2022-09-04T14:35:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067311599999999999 , 2022-09-04T14:35:02.575 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:02.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834326+0000", , 2022-09-04T14:35:02.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:35:02.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:35:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.777357+0000", , 2022-09-04T14:35:02.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1764e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:02.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.786750+0000", , 2022-09-04T14:35:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070554800000000003 Ignoring malparsed backtrace: 2022-09-04T14:35:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:02.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:02.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.022799+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:02.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:35:02.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.908179+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:02.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.937638+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:02.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:02.785 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:02.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:02.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:02.813 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:02.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:02.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:02.844 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:02.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:02.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:02.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:02.875 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:03.108 INFO:tasks.rados.rados.0.smithi163.stdout:308: done (2 left) , 2022-09-04T14:35:03.108 INFO:tasks.rados.rados.0.smithi163.stdout:310: done (1 left) , 2022-09-04T14:35:03.109 INFO:tasks.rados.rados.0.smithi163.stdout:311: snap_remove snap 23 Ignoring malparsed backtrace: 2022-09-04T14:35:03.332 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:35:03.332 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:03.332 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:03.356 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:03.357 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:03.357 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:03.385 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:03.385 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:03.385 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:35:03.409 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:03.409 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:35:03.409 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:03.434 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.2855000000000001e-05 , 2022-09-04T14:35:03.434 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:03.434 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:03.458 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.7930000000000003e-06 , 2022-09-04T14:35:03.458 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:03.458 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:03.482 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:35:03.482 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:03.482 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:03.507 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833036+0000", , 2022-09-04T14:35:03.507 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4566000000000005e-05 , 2022-09-04T14:35:03.507 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:03.531 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834020+0000", , 2022-09-04T14:35:03.531 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.6729999999999998e-06 , 2022-09-04T14:35:03.531 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:03.555 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0039679629999999997, , 2022-09-04T14:35:03.555 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:35:03.555 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:35:03.579 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:35:03.579 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686080+0000", , 2022-09-04T14:35:03.579 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0021525260000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:03.603 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:03.603 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686475+0000", , 2022-09-04T14:35:03.603 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.4739999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:03.627 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 12.222506437, , 2022-09-04T14:35:03.627 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0048458909999999997, , 2022-09-04T14:35:03.628 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:03.652 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:03.652 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:35:03.652 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.069456+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:03.675 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:03.676 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:03.676 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.073019+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:04.174 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.035869856999999998 , 2022-09-04T14:35:04.174 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:04.175 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:04.210 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.6820000000000003e-06 , 2022-09-04T14:35:04.210 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:04.211 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:04.244 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:35:04.245 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:04.245 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:04.290 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.835507+0000", , 2022-09-04T14:35:04.290 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.001094898 , 2022-09-04T14:35:04.291 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:04.313 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:24.598551+0000", , 2022-09-04T14:35:04.314 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.0975000000000001e-05 , 2022-09-04T14:35:04.314 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:04.337 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0064595980000000004, , 2022-09-04T14:35:04.337 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:35:04.337 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:35:04.361 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:35:04.362 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843624+0000", , 2022-09-04T14:35:04.362 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.5616999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:04.387 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:04.387 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684810+0000", , 2022-09-04T14:35:04.387 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4609999999999996e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:04.411 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 29.138540235000001, , 2022-09-04T14:35:04.411 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.25616860400000002, , 2022-09-04T14:35:04.412 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:04.435 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:04.436 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:35:04.436 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.968564+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:04.461 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:04.461 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:04.461 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.972473+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:04.485 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:46.966547+0000", , 2022-09-04T14:35:04.486 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 17.194025951, , 2022-09-04T14:35:04.486 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0046854039999999998, Ignoring malparsed backtrace: 2022-09-04T14:35:04.510 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:04.510 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:04.510 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:35:04.534 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:04.534 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:04.534 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:35:04.559 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.0s4 86/33 ECSubWrite(tid=128, reqid=client.4557.0:793, at_version=0'0, trim_to=0'0, roll_forward_to=86'71))", , 2022-09-04T14:35:04.559 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:35:03.105466+0000", , 2022-09-04T14:35:04.559 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 1.055107483, Ignoring malparsed backtrace: 2022-09-04T14:35:04.856 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} , 2022-09-04T14:35:04.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops , 2022-09-04T14:35:04.889+0000 7f91016aa700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 39166) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:05.015 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:35:05.016 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:05.016 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:05.040 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.3050000000000003e-05 , 2022-09-04T14:35:05.040 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:05.041 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:05.066 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.1620999999999999e-05 , 2022-09-04T14:35:05.066 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:05.067 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:05.099 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816711+0000", , 2022-09-04T14:35:05.099 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:35:05.100 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:05.132 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831226+0000", , 2022-09-04T14:35:05.133 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4857999999999995e-05 , 2022-09-04T14:35:05.133 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:05.157 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833638+0000", , 2022-09-04T14:35:05.157 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00033025399999999999 , 2022-09-04T14:35:05.157 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:05.181 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:35:05.182 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.832850+0000", , 2022-09-04T14:35:05.182 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:05.206 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:35:05.206 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834407+0000", , 2022-09-04T14:35:05.207 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.5741999999999993e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:05.230 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:05.230 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.972557+0000", , 2022-09-04T14:35:05.230 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2726e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:05.254 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:05.255 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:35:05.255 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:44.960854+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:05.279 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:05.279 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:35:05.280 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:46.966824+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:05.303 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:05.304 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:05.304 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.071289+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:05.328 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:05.328 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:05.328 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:35:05.352 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:05.352 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:05.352 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:05.377 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:05.377 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:05.377 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:35:06.119 INFO:tasks.rados.rados.0.smithi163.stdout:320: done (6 left) , 2022-09-04T14:35:06.119 INFO:tasks.rados.rados.0.smithi163.stdout:321: rollback oid 35 current snap is 26 , 2022-09-04T14:35:06.119 INFO:tasks.rados.rados.0.smithi163.stdout:rollback oid 35 to 25 Ignoring malparsed backtrace: 2022-09-04T14:35:07.111+0000 7f8152796700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset e=[]:{e=[e,c,b,9]} for object 3:0dcdd893:::smithi16339483-22:8 , 2022-09-04T14:35:07.114+0000 7f793eff5700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 27=[]:{17=[17,14,12,e,c],27=[27,25,23,22,1f]} for object 3:a9d45bed:::smithi16339483-27:2 , 2022-09-04T14:35:07.115+0000 7faa2b696700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 33=[]:{2b=[2b,27,23,14],33=[33,2f]} for object 3:70fb2152:::smithi16339483-29:9 Ignoring malparsed backtrace: 2022-09-04T14:35:08.125 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.393234429000003, , 2022-09-04T14:35:08.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013886004, , 2022-09-04T14:35:08.126 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:08.148 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 55.393207547000003, , 2022-09-04T14:35:08.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023000298999999998, , 2022-09-04T14:35:08.149 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:08.172 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.172 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:35:08.172 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.754545+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:08.195 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.195 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:08.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776574+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:08.220 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.221 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:08.221 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776726+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:08.247 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:08.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.780473+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:08.273 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:08.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.790690+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:08.305 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:08.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.795098+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:08.332 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:08.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.807732+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:08.357 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:08.383 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:08.409 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.410 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:35:08.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:08.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:08.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:35:08.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:08.463 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:08.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010484019999999999 , 2022-09-04T14:35:08.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:08.488 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:08.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00049445999999999995 , 2022-09-04T14:35:08.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:08.512 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:08.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3847000000000003e-05 , 2022-09-04T14:35:08.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:08.539 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:08.865 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 , 2022-09-04T14:35:08.865 INFO:teuthology.orchestra.run.smithi163.stdout:} , 2022-09-04T14:35:08.866+0000 7fc80fcb9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 39177) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:09.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:09.069 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:09.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:09.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:09.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:09.096 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:09.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6909999999999996e-06 , 2022-09-04T14:35:09.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:09.124 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:09.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0159000000000001e-05 , 2022-09-04T14:35:09.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:09.152 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:09.176 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.834310+0000", , 2022-09-04T14:35:09.176 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:09.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:09.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.835207+0000", , 2022-09-04T14:35:09.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0599999999999998e-06 , 2022-09-04T14:35:09.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:09.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.840259+0000", , 2022-09-04T14:35:09.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014308070000000001 , 2022-09-04T14:35:09.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:09.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:09.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.682716+0000", , 2022-09-04T14:35:09.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:09.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:09.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.682866+0000", , 2022-09-04T14:35:09.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9900000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:09.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.089383+0000", , 2022-09-04T14:35:09.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000218054 Ignoring malparsed backtrace: 2022-09-04T14:35:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:09.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:09.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.055362+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:09.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.058559+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:09.380 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:09.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.063533+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:09.404 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.069997+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:09.427 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:59.226201+0000", , 2022-09-04T14:35:09.427 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.8131991850000002, , 2022-09-04T14:35:09.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.113233421, Ignoring malparsed backtrace: osd.12: osd_enable_op_tracker = 'true' , Error ENXIO: problem getting command descriptions from osd.13 , osd.14: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:35:09.950 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7362999999999996e-05 , 2022-09-04T14:35:09.950 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:09.950 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:09.984 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.802650+0000", , 2022-09-04T14:35:09.984 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:09.984 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:10.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:19.063973+0000", , 2022-09-04T14:35:10.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4679e-05 , 2022-09-04T14:35:10.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:10.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.809731+0000", , 2022-09-04T14:35:10.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1881999999999997e-05 , 2022-09-04T14:35:10.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:10.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:10.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.782609+0000", , 2022-09-04T14:35:10.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:10.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:10.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683472+0000", , 2022-09-04T14:35:10.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1270000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:10.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:10.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.684395+0000", , 2022-09-04T14:35:10.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3733999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:10.139 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:10.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:10.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.699445+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:10.164 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:10.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:10.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.733314+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:10.189 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:10.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:10.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:35.024186+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:10.217 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:10.218 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:10.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:10.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:10.245 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:10.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:10.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:10.271 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:10.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:35:10.296 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:10.297 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:10.297 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:10.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9177e-05 , 2022-09-04T14:35:10.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:10.327 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:10.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:35:10.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.828465+0000", , 2022-09-04T14:35:10.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:10.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:35:10.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.835930+0000", , 2022-09-04T14:35:10.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2760000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:10.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:10.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.831773+0000", , 2022-09-04T14:35:10.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1087999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:10.908 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:10.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:35:10.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.862141+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:10.937 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:10.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:10.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.683032+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:10.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:10.969 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:10.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:35:10.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:10.993 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:10.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:11.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.018 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:35:11.042 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 69/52 ECSubWriteReply(tid=54, last_complete=69'30, committed=1, applied=1))", , 2022-09-04T14:35:11.042 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:45.965465+0000", , 2022-09-04T14:35:11.042 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.848718131999998, Ignoring malparsed backtrace: 2022-09-04T14:35:11.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014546599999999999 , 2022-09-04T14:35:11.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.066 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:11.090 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.090 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s5 74/33 ECSubWrite(tid=46, reqid=client.4557.0:638, at_version=74'31, trim_to=0'0, roll_forward_to=63'27))", , 2022-09-04T14:35:11.090 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:51.077396+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:11.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077967+0000", , 2022-09-04T14:35:11.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0755e-05 , 2022-09-04T14:35:11.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.079101+0000", , 2022-09-04T14:35:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012972400000000001 , 2022-09-04T14:35:11.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:11.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.162 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.162 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.7s3 92/33 ECSubWrite(tid=260, reqid=client.4557.0:860, at_version=92'74, trim_to=0'0, roll_forward_to=78'72))", Ignoring malparsed backtrace: 2022-09-04T14:35:11.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:35:11.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.131632+0000", , 2022-09-04T14:35:11.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000940766 Ignoring malparsed backtrace: 2022-09-04T14:35:11.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:35:11.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.722763+0000", , 2022-09-04T14:35:11.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:11.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:35:11.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.724126+0000", , 2022-09-04T14:35:11.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.725e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:11.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:35:11.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.737326+0000", , 2022-09-04T14:35:11.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076465750000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:11.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:35:11.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.738323+0000", , 2022-09-04T14:35:11.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083854519999999998 Ignoring malparsed backtrace: 2022-09-04T14:35:11.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:11.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.753392+0000", , 2022-09-04T14:35:11.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4231e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:11.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:11.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.762617+0000", , 2022-09-04T14:35:11.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00086305099999999999 Ignoring malparsed backtrace: 2022-09-04T14:35:11.816 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:35:11.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.830556+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:11.839 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:35:11.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.860982+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:11.863 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:11.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.798464+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:11.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.887 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:11.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.910 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:11.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.933 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:11.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:11.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6229999999999996e-06 , 2022-09-04T14:35:11.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.957 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:11.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2159999999999997e-06 , 2022-09-04T14:35:11.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:11.981 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:12.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048082000000000002 , 2022-09-04T14:35:12.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:12.005 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:12.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.103146+0000", , 2022-09-04T14:35:12.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:12.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:12.541 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:35:12.542 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:35:12.543+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:14.153 INFO:tasks.rados.rados.0.smithi163.stdout:355: read oid 28 snap -1 , 2022-09-04T14:35:14.154 INFO:tasks.rados.rados.0.smithi163.stdout:355: expect (ObjNum 37 snap 0 seq_num 37) , 2022-09-04T14:35:14.154 INFO:tasks.rados.rados.0.smithi163.stdout:356: rollback oid 3 current snap is 31 Ignoring malparsed backtrace: 2022-09-04T14:35:14.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0074087370000000003, , 2022-09-04T14:35:14.762 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:14.762 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:35:14.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:14.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.827184+0000", , 2022-09-04T14:35:14.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00077420899999999999 Ignoring malparsed backtrace: 2022-09-04T14:35:14.816 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:14.816 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828897+0000", , 2022-09-04T14:35:14.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:14.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:35:14.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.829802+0000", , 2022-09-04T14:35:14.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105164 Ignoring malparsed backtrace: 2022-09-04T14:35:14.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:35:14.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.842198+0000", , 2022-09-04T14:35:14.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7017e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:14.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:35:14.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.829144+0000", , 2022-09-04T14:35:14.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0094134570000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:14.938 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:14.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:14.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.829704+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:14.967 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:35:14.967 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:35:14.967 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:35:14.990 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:35:14.991 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:35:14.991 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:35:15.014 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:35:15.014 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:35:15.014 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:35:15.037 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:35:15.037 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:35:15.037 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:35:15.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.060 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:15.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.108 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:15.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.137 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:15.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.166 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:35:15.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.188 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2022-09-04T14:35:15.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.211 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2022-09-04T14:35:15.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.683 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:15.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.718 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:15.748 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:62 3.ds0 3:b87fd6ef:::smithi16339483-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:35:15.748 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.749505+0000", , 2022-09-04T14:35:15.749 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.909985100999997, Ignoring malparsed backtrace: 2022-09-04T14:35:15.783 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:73 3.ds0 3:b1abc525:::smithi16339483-25:head [append 0~409600,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:35:15.784 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.759235+0000", , 2022-09-04T14:35:15.784 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.900255043999998, Ignoring malparsed backtrace: 2022-09-04T14:35:15.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.808 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:35:15.808 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:15.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00133603 , 2022-09-04T14:35:15.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.831 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:15.873 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:15.873 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:109 3.as0 3:5170752b:::smithi16339483-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:35:15.874 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.799999+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:15.910 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 110 , 2022-09-04T14:35:15.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.911 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:15.938 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 111 , 2022-09-04T14:35:15.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:15.939 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:15.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858781+0000", , 2022-09-04T14:35:15.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030874522000000001 , 2022-09-04T14:35:15.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:15.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.794945+0000", , 2022-09-04T14:35:15.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014915832 , 2022-09-04T14:35:15.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801246+0000", , 2022-09-04T14:35:16.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0057189500000000004 , 2022-09-04T14:35:16.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.543777+0000", , 2022-09-04T14:35:16.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8559000000000005e-05 , 2022-09-04T14:35:16.034 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:16.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:16.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.091533+0000", , 2022-09-04T14:35:16.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:16.080 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:16.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:16.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:14.246847+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:16.170+0000 7efdcd395700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2f=[]:{2f=[2f]} for object 3:b771c3b4:::smithi16339483-5:9 , 2022-09-04T14:35:16.172+0000 7f814e78e700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset e=[]:{e=[e,c,b,9]} for object 3:0dcdd893:::smithi16339483-22:8 , 2022-09-04T14:35:16.175 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub Ignoring malparsed backtrace: 2022-09-04T14:35:16.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.531 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:16.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:16.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.557 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:16.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:16.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.588 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:16.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:35:16.614 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:100 3.5s0 3:a5dd5b1a:::smithi16339483-34:head [append 0~557056,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:35:16.614 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.788724+0000", , 2022-09-04T14:35:16.614 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.719844653000003, Ignoring malparsed backtrace: 2022-09-04T14:35:16.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.637 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:35:16.638 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:16.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.660 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:35:16.661 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:16.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0299999999999999e-06 , 2022-09-04T14:35:16.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.684 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:16.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5809999999999998e-06 , 2022-09-04T14:35:16.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.707 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:16.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9212999999999998e-05 , 2022-09-04T14:35:16.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:16.730 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:16.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784565+0000", , 2022-09-04T14:35:16.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:35:16.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784898+0000", , 2022-09-04T14:35:16.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0599999999999997e-06 , 2022-09-04T14:35:16.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.804047+0000", , 2022-09-04T14:35:16.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020287489999999998 , 2022-09-04T14:35:16.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.834482+0000", , 2022-09-04T14:35:16.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7153e-05 , 2022-09-04T14:35:16.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.834488+0000", , 2022-09-04T14:35:16.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2362999999999999e-05 , 2022-09-04T14:35:16.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.910147+0000", , 2022-09-04T14:35:16.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115427 , 2022-09-04T14:35:16.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:16.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:16.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069387+0000", , 2022-09-04T14:35:16.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2960000000000004e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:16.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:16.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.126923+0000", , 2022-09-04T14:35:16.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000153172 Ignoring malparsed backtrace: 2022-09-04T14:35:17.435 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:35:17.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749815+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:17.476 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:35:17.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749834+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:17.503 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:35:17.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.002446+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:17.526 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:17.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.006078+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:17.548 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:35:17.549 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:17.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:17.571 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:17.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:17.594 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:35:17.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:17.616 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:17.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2277e-05 , 2022-09-04T14:35:17.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:17.640 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:17.662 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:17.662 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:572 3.cs0 3:36aacf6f:::smithi16339483-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", , 2022-09-04T14:35:17.663 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:44.949092+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:17.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.951124+0000", , 2022-09-04T14:35:17.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9839999999999998e-06 , 2022-09-04T14:35:17.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:17.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.951156+0000", , 2022-09-04T14:35:17.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.984e-06 , 2022-09-04T14:35:17.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069420+0000", , 2022-09-04T14:35:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2003000000000003e-05 , 2022-09-04T14:35:17.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071198+0000", , 2022-09-04T14:35:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00031387100000000002 , 2022-09-04T14:35:17.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0081005939999999992, , 2022-09-04T14:35:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:17.778 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:35:17.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:35:17.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.175753+0000", , 2022-09-04T14:35:17.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6483999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:18.419+0000 7f7962451700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 38719) UID: 0 , 2022-09-04T14:35:18.462+0000 7faa2b696700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 33=[]:{2b=[2b,27,23,14],33=[33,2f]} for object 3:70fb2152:::smithi16339483-29:9 , 2022-09-04T14:35:18.521+0000 7f91016aa700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 39166) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:20.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.828464+0000", , 2022-09-04T14:35:20.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:35:20.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:20.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.835958+0000", , 2022-09-04T14:35:20.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8277999999999999e-05 , 2022-09-04T14:35:20.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:20.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.831989+0000", , 2022-09-04T14:35:20.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021620799999999999 , 2022-09-04T14:35:20.134 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:20.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:20.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.862140+0000", , 2022-09-04T14:35:20.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:35:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.703649+0000", , 2022-09-04T14:35:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020617862000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:20.201 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:20.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:20.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910371+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:20.223 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:20.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:35:20.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910840+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:20.245 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:20.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:20.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.965253+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:20.267 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:35:20.267 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:20.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:20.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:20.290 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:20.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:35:20.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:20.312 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:35:20.312 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:20.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8885999999999998e-05 , 2022-09-04T14:35:20.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:20.335 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:20.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1438999999999999e-05 , 2022-09-04T14:35:20.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:20.357 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:20.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:35:20.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:20.379 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:20.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.131667+0000", , 2022-09-04T14:35:20.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5179000000000003e-05 , 2022-09-04T14:35:20.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 , 2022-09-04T14:35:20.795 INFO:teuthology.orchestra.run.smithi038.stdout:} , 2022-09-04T14:35:20.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops Ignoring malparsed backtrace: 2022-09-04T14:35:20.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:20.966 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:35:20.967 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:21.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.578e-06 , 2022-09-04T14:35:21.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:21.009 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:21.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5049999999999999e-06 , 2022-09-04T14:35:21.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:21.040 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:21.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010776 , 2022-09-04T14:35:21.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:21.070 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:21.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0911e-05 , 2022-09-04T14:35:21.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:21.094 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:21.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014114999999999999 , 2022-09-04T14:35:21.116 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:35:21.116 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:35:21.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:35:21.138 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", , 2022-09-04T14:35:21.138 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:35:21.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.780166+0000", , 2022-09-04T14:35:21.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.426e-06 , 2022-09-04T14:35:21.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:21.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.932173+0000", , 2022-09-04T14:35:21.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022800191000000001 , 2022-09-04T14:35:21.183 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:21.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:21.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916278+0000", , 2022-09-04T14:35:21.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:35:21.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:35:21.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.978476+0000", , 2022-09-04T14:35:21.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014725059999999999 Ignoring malparsed backtrace: 2022-09-04T14:35:21.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:21.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:59.080489+0000", , 2022-09-04T14:35:21.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028804630000000002 Ignoring malparsed backtrace: 2022-09-04T14:35:21.272 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:21.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:35:21.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.101204+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:21.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:21.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.101246+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:21.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:21.324 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:21.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:35:21.516 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 , 2022-09-04T14:35:21.516 INFO:teuthology.orchestra.run.smithi038.stdout:} , 2022-09-04T14:35:21.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:35:21.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:35:21.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.773854+0000", , 2022-09-04T14:35:21.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:21.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:21.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682571+0000", , 2022-09-04T14:35:21.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9460000000000001e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.683166+0000", , 2022-09-04T14:35:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8107e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:35:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.157225+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.909309+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910830+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:35:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:22.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:22.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.073 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:22.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4573000000000001e-05 , 2022-09-04T14:35:22.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.096 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:22.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2776e-05 , 2022-09-04T14:35:22.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.120 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:22.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:10.127196+0000", , 2022-09-04T14:35:22.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:22.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:22.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:16.171715+0000", , 2022-09-04T14:35:22.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:22.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:22.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:16.172762+0000", , 2022-09-04T14:35:22.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7199e-05 , 2022-09-04T14:35:22.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5858999999999997e-05 , 2022-09-04T14:35:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6794999999999997e-05 , 2022-09-04T14:35:22.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.686 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:22.708 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.708 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s4 47/33 ECSubWrite(tid=20, reqid=client.4557.0:299, at_version=47'14, trim_to=0'0, roll_forward_to=37'11))", , 2022-09-04T14:35:22.708 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:23.833106+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:22.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833908+0000", , 2022-09-04T14:35:22.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6799999999999998e-05 , 2022-09-04T14:35:22.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:22.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834565+0000", , 2022-09-04T14:35:22.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3308999999999997e-05 , 2022-09-04T14:35:22.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:22.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.776 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.776 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s4 51/33 ECSubWrite(tid=26, reqid=client.4557.0:340, at_version=0'0, trim_to=0'0, roll_forward_to=51'19))", Ignoring malparsed backtrace: 2022-09-04T14:35:22.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:35:22.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.022819+0000", , 2022-09-04T14:35:22.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9885e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:22.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:22.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.908328+0000", , 2022-09-04T14:35:22.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014860700000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:22.845 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:35:22.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.845 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:22.867 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:35:22.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916307+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:22.889 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:22.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954885+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:22.912 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:35:22.912 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:35:22.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:22.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.935 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:22.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:22.957 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:22.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:35:22.979 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:35:22.979 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:35:22.980 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: osd.0: osd_enable_op_tracker = 'false' , osd.1: osd_enable_op_tracker = 'false' , osd.2: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:35:24.213 INFO:tasks.rados.rados.0.smithi163.stdout:396: writing smithi16339483-38 from 2801664 to 2949120 tid 1 , 2022-09-04T14:35:24.214 INFO:tasks.rados.rados.0.smithi163.stdout:397: rollback oid 18 current snap is 34 , 2022-09-04T14:35:24.214 INFO:tasks.rados.rados.0.smithi163.stdout:rollback oid 18 to 32 Ignoring malparsed backtrace: 2022-09-04T14:35:25.464+0000 7f4dfdc57700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{c=[c],15=[15,14,12,11,e],1e=[1e,1b,17],2b=[27,23],43=[42]} for object 3:38890410:::smithi16339483-19:9 , 2022-09-04T14:35:25.486 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} , 2022-09-04T14:35:25.508 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} Ignoring malparsed backtrace: 2022-09-04T14:35:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.740723+0000", , 2022-09-04T14:35:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011178647 , 2022-09-04T14:35:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:35:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 80 , 2022-09-04T14:35:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:35:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 81 , 2022-09-04T14:35:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.789240+0000", , 2022-09-04T14:35:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051742499999999996 Ignoring malparsed backtrace: 2022-09-04T14:35:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.789284+0000", , 2022-09-04T14:35:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5329999999999996e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:25.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:25.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.789310+0000", , 2022-09-04T14:35:25.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1014e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:25.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:25.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.865961+0000", , 2022-09-04T14:35:25.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000196839 Ignoring malparsed backtrace: 2022-09-04T14:35:25.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:35:25.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.978077+0000", , 2022-09-04T14:35:25.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2662000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:25.813 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:35:25.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783674+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784657+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:25.868 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:25.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.785157+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:25.891 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:35:25.891 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:25.916 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:35:25.916 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:25.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:25.940 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:35:25.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:25.967 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:25.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:25.992 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:25.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: osd.2: osd_enable_op_tracker = 'true' , osd.3: osd_enable_op_tracker = 'true' , osd.4: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:35:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749820+0000", , 2022-09-04T14:35:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5460000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749839+0000", , 2022-09-04T14:35:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1330000000000001e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.002527+0000", , 2022-09-04T14:35:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0981000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:35:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.006152+0000", , 2022-09-04T14:35:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4447000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:35:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.815258+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:26.675 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:26.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:26.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816639+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:26.699 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:26.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:26.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.843805+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:26.722 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:35:26.722 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:35:26.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:26.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:26.745 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:26.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", Ignoring malparsed backtrace: 2022-09-04T14:35:26.768 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:35:26.768 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:35:26.768 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:35:26.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9339999999999995e-06 , 2022-09-04T14:35:26.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:26.792 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:26.816 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.822e-06 , 2022-09-04T14:35:26.816 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:26.816 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:26.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0192e-05 , 2022-09-04T14:35:26.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:26.840 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:26.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080663100000000004 , 2022-09-04T14:35:26.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:26.863 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:26.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119125+0000", , 2022-09-04T14:35:26.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:35:26.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:26.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.175762+0000", , 2022-09-04T14:35:26.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5300000000000002e-06 , 2022-09-04T14:35:26.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:27.350 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.20437936200000001 , 2022-09-04T14:35:27.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:27.351 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:27.376 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033857700000000001 , 2022-09-04T14:35:27.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:27.376 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:27.399 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2769999999999996e-06 , 2022-09-04T14:35:27.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:27.400 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:27.422 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:35:27.423 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:27.423 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:27.447 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.679993+0000", , 2022-09-04T14:35:27.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2919999999999998e-06 , 2022-09-04T14:35:27.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:27.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680048+0000", , 2022-09-04T14:35:27.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8559999999999998e-06 , 2022-09-04T14:35:27.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:27.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.682543+0000", , 2022-09-04T14:35:27.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112991 , 2022-09-04T14:35:27.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:27.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.083498+0000", , 2022-09-04T14:35:27.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019956999999999999 , 2022-09-04T14:35:27.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:27.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.085074+0000", , 2022-09-04T14:35:27.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110553 , 2022-09-04T14:35:27.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:27.565 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:27.565 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046751+0000", , 2022-09-04T14:35:27.566 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:27.590 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:27.590 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046906+0000", , 2022-09-04T14:35:27.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107398 Ignoring malparsed backtrace: 2022-09-04T14:35:27.614 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:27.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:35:27.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.052267+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:27.638 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:35:27.639 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:35:27.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:27.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:27.663 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:27.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:27.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:27.687 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:27.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:35:27.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:27.711 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:27.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2022-09-04T14:35:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.733283+0000", , 2022-09-04T14:35:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035355120000000002 Ignoring malparsed backtrace: 2022-09-04T14:35:28.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:28.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.734116+0000", , 2022-09-04T14:35:28.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036561300000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:28.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:28.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.768228+0000", , 2022-09-04T14:35:28.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030486430000000002 Ignoring malparsed backtrace: 2022-09-04T14:35:28.217 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:28.217 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.788929+0000", , 2022-09-04T14:35:28.217 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000237197 Ignoring malparsed backtrace: 2022-09-04T14:35:28.240 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:28.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778221+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:28.263 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:35:28.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778538+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:28.287 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:35:28.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778586+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:28.310 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:28.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.783172+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:28.333 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:28.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.783293+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:28.357 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:28.357 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.832480+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:28.380 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:28.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.867405+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:28.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:28.405 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:28.405 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:28.428 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:35:28.428 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:35:28.429 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:35:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:35:28.454 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:35:28.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:28.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:28.478 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:28.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:28.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:28.501 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:28.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024222800000000001 , 2022-09-04T14:35:28.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:28.525 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:29.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight , 2022-09-04T14:35:29.040+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 , 2022-09-04T14:35:29.141+0000 7f4e208b2700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 38721) UID: 0 Ignoring malparsed backtrace: osd.2: osd_enable_op_tracker = 'true' , osd.3: osd_enable_op_tracker = 'true' , osd.4: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:35:31.733 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:31.733 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:35:31.733 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.785691+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:31.757 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:31.757 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:31.757 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:31.782 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:31.782 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:31.782 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:31.806 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:31.806 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:31.806 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:31.831 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:35:31.831 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:31.831 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:31.858 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.6230999999999998e-05 , 2022-09-04T14:35:31.858 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:31.858 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:31.887 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.5543e-05 , 2022-09-04T14:35:31.888 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:31.888 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:31.914 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.683921+0000", , 2022-09-04T14:35:31.914 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:35:31.914 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:31.939 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686409+0000", , 2022-09-04T14:35:31.939 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00031561300000000002 , 2022-09-04T14:35:31.939 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:31.965 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:34.350153+0000", , 2022-09-04T14:35:31.965 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.7318000000000002e-05 , 2022-09-04T14:35:31.965 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:31.989 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:35:31.989 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.069389+0000", , 2022-09-04T14:35:31.989 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:32.014 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:35:32.014 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.072872+0000", , 2022-09-04T14:35:32.014 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.2235999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:32.039 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:32.039 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:09.132484+0000", , 2022-09-04T14:35:32.039 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.4035e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:32.064 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:32.064 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:35:32.064 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192188+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:32.088 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:32.089 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:35:32.089 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194414+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:32.600 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:32.600 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:32.601 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.809732+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:32.629 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:32.629 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:32.629 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.814680+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:32.652 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:35:32.652 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:32.652 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:32.675 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:32.675 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:32.676 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:32.698 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:32.698 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:32.698 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:35:32.721 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:32.721 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:35:32.721 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:32.744 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.5616999999999999e-05 , 2022-09-04T14:35:32.744 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:32.744 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:32.767 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4609999999999996e-06 , 2022-09-04T14:35:32.767 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:32.767 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:32.789 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:35:32.789 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:32.790 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:32.812 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.968564+0000", , 2022-09-04T14:35:32.813 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.8151000000000001e-05 , 2022-09-04T14:35:32.813 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:32.835 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.972473+0000", , 2022-09-04T14:35:32.836 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000829015 , 2022-09-04T14:35:32.836 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:32.859 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0046854039999999998, , 2022-09-04T14:35:32.859 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:35:32.860 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:35:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:35:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.070508+0000", , 2022-09-04T14:35:32.882 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.2145999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.075878+0000", , 2022-09-04T14:35:32.905 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.6610000000000006e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 29.475921125999999, , 2022-09-04T14:35:32.927 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0082416109999999994, , 2022-09-04T14:35:32.928 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:33.399 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.bs6 42/33 ECSubWrite(tid=58, reqid=client.4557.0:196, at_version=42'10, trim_to=0'0, roll_forward_to=36'7))", , 2022-09-04T14:35:33.399 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:18.783212+0000", , 2022-09-04T14:35:33.400 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 74.609963360999998, Ignoring malparsed backtrace: 2022-09-04T14:35:33.432 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2748999999999999e-05 , 2022-09-04T14:35:33.432 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:33.433 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:33.464 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00011767700000000001 , 2022-09-04T14:35:33.464 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:33.464 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:33.487 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.487 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.1s2 47/41 ECSubWrite(tid=79, reqid=client.4557.0:286, at_version=47'12, trim_to=0'0, roll_forward_to=44'8))", , 2022-09-04T14:35:33.487 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:23.816711+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:33.509 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831107+0000", , 2022-09-04T14:35:33.510 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.5099999999999999e-05 , 2022-09-04T14:35:33.510 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:33.532 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833234+0000", , 2022-09-04T14:35:33.532 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000152607 , 2022-09-04T14:35:33.532 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:33.557 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:33.557 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.558 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/33 ECSubWrite(tid=85, reqid=client.4557.0:293, at_version=47'28, trim_to=0'0, roll_forward_to=47'25))", Ignoring malparsed backtrace: 2022-09-04T14:35:33.585 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:35:33.586 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834189+0000", , 2022-09-04T14:35:33.586 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.8318999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:33.618 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:35:33.619 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971532+0000", , 2022-09-04T14:35:33.619 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.1555e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:33.657 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:35:33.657 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.979149+0000", , 2022-09-04T14:35:33.657 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00566338 Ignoring malparsed backtrace: 2022-09-04T14:35:33.695 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.695 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:35:33.696 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.063495+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:33.735 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.736 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:35:33.736 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.076006+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:33.771 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.771 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:35:33.772 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.089513+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:33.807 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:33.808 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.808 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:33.840 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:33.840 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.840 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:33.871 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:33.872 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:33.872 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:35.547+0000 7efdcd395700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2f=[]:{2f=[2f]} for object 3:b771c3b4:::smithi16339483-5:9 , 2022-09-04T14:35:35.592+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:35:35.633 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:35:36.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.604 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:36.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:36.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.631 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:36.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:36.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.660 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:36.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:36.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.687 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:36.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:36.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.715 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:36.715 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:36.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:36.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.743 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:36.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:36.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.770 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:36.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0970000000000002e-06 , 2022-09-04T14:35:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.793 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.899e-06 , 2022-09-04T14:35:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.816 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9789000000000006e-05 , 2022-09-04T14:35:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.839 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.075144469000000005 , 2022-09-04T14:35:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:36.863 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:36.885 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.792299+0000", , 2022-09-04T14:35:36.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 , 2022-09-04T14:35:36.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:36.908 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.913060+0000", , 2022-09-04T14:35:36.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012248700000000001 , 2022-09-04T14:35:36.909 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:36.932 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062107810999999999, , 2022-09-04T14:35:36.932 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:35:36.932 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:35:36.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:36.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.116484+0000", , 2022-09-04T14:35:36.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:36.979 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:36.979 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.116538+0000", , 2022-09-04T14:35:36.979 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:37.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:35:37.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.263796+0000", , 2022-09-04T14:35:37.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024222800000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:37.261 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight , 2022-09-04T14:35:37.316+0000 7f84b1586700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 39181) UID: 0 , 2022-09-04T14:35:37.324 INFO:teuthology.orchestra.run.smithi163.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:35:37.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.781058+0000", , 2022-09-04T14:35:37.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4619e-05 , 2022-09-04T14:35:37.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:37.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012638956, , 2022-09-04T14:35:37.567 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:35:37.567 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:35:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:35:37.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.832930+0000", , 2022-09-04T14:35:37.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2645000000000005e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:37.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833196+0000", , 2022-09-04T14:35:37.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5059999999999997e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 73.665906957000004, , 2022-09-04T14:35:37.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011243863999999999, , 2022-09-04T14:35:37.642 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:37.664 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:37.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:35:37.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.835202+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:37.686 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:37.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:37.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.838829+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:37.709 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:58.090349+0000", , 2022-09-04T14:35:37.709 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.409868076000002, , 2022-09-04T14:35:37.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0073672010000000003, Ignoring malparsed backtrace: 2022-09-04T14:35:37.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:37.731 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:37.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:35:37.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:37.755 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:37.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:37.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:37.777 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:37.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:35:37.799 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:37.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:37.800 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:37.822 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9580000000000007e-06 , 2022-09-04T14:35:37.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:37.822 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:37.845 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012516129999999999 , 2022-09-04T14:35:37.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:37.845 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:37.871 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:22.192220+0000", , 2022-09-04T14:35:37.871 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:37.871 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:38.338 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.795154+0000", , 2022-09-04T14:35:38.339 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:38.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:38.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.802784+0000", , 2022-09-04T14:35:38.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5069999999999997e-06 , 2022-09-04T14:35:38.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:38.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:19.064416+0000", , 2022-09-04T14:35:38.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3440000000000002e-05 , 2022-09-04T14:35:38.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:38.568 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.780204+0000", , 2022-09-04T14:35:38.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.782680+0000", , 2022-09-04T14:35:38.591 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4864e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.684394+0000", , 2022-09-04T14:35:38.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4196999999999995e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:38.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.694460+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:38.657 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:38.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:38.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.733314+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:38.679 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:38.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:38.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:41.957889+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:38.702 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:38.702 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:38.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:38.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:38.728 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:38.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:38.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:38.750 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:38.751 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:35:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:38.773 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:38.774 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:38.795 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2636e-05 , 2022-09-04T14:35:38.796 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:38.796 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:38.818 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4749999999999996e-06 , 2022-09-04T14:35:38.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:38.818 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:38.839+0000 7f91016aa700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 39166) UID: 0 , 2022-09-04T14:35:38.740+0000 7f403142b700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 , 2022-09-04T14:35:38.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight Ignoring malparsed backtrace: 2022-09-04T14:35:40.821 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:35:40.822 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:35:40.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight Ignoring malparsed backtrace: 2022-09-04T14:35:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 6 , 2022-09-04T14:35:42.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring , 2022-09-04T14:35:42.478+0000 7f4052883700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38941) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:42.584 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:42.585 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:275 3.fs0 3:fb230078:::smithi16339483-17:head [append 0~376832] snapc 9=[9,8,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", , 2022-09-04T14:35:42.585 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:21.828486+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:42.609 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 276 , 2022-09-04T14:35:42.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:42.609 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:42.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828939+0000", , 2022-09-04T14:35:42.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:35:42.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:42.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.813965+0000", , 2022-09-04T14:35:42.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5882999999999999e-05 , 2022-09-04T14:35:42.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:42.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.829848+0000", , 2022-09-04T14:35:42.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000120616 , 2022-09-04T14:35:42.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:42.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:42.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.829749+0000", , 2022-09-04T14:35:42.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.741e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:42.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:42.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.773951+0000", , 2022-09-04T14:35:42.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8943000000000005e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:42.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:42.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.778444+0000", , 2022-09-04T14:35:42.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012493899999999999 Ignoring malparsed backtrace: 2022-09-04T14:35:42.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:42.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.779312+0000", , 2022-09-04T14:35:42.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7971e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:42.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:35:42.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:29.679887+0000", , 2022-09-04T14:35:42.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013270099999999999 Ignoring malparsed backtrace: 2022-09-04T14:35:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:29.679689+0000", , 2022-09-04T14:35:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0116857909999999 Ignoring malparsed backtrace: 2022-09-04T14:35:42.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:35:42.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.090019+0000", , 2022-09-04T14:35:42.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0035327829999999998 Ignoring malparsed backtrace: 2022-09-04T14:35:42.872 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:35:42.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:42.873 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:42.895 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:42.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:42.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071657+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:42.917 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:42.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", , 2022-09-04T14:35:42.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:59.049166+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:42.940 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:42.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", , 2022-09-04T14:35:42.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.166730+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:42.963 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:42.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", , 2022-09-04T14:35:42.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.166770+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:43.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:35:43.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.721955+0000", , 2022-09-04T14:35:43.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:35:43.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:43.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.725452+0000", , 2022-09-04T14:35:43.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003179299 Ignoring malparsed backtrace: 2022-09-04T14:35:43.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:43.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.736030+0000", , 2022-09-04T14:35:43.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021720519999999998 Ignoring malparsed backtrace: 2022-09-04T14:35:43.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:43.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.762515+0000", , 2022-09-04T14:35:43.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3252999999999996e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:43.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:35:43.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.785317+0000", , 2022-09-04T14:35:43.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4043999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:43.549 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:35:43.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.759688+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:43.573 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:43.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.787084+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:43.597 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:35:43.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.845754+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:43.621 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:35:43.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.847412+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:35:43.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:43.670 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:43.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:43.694 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for blocked object", Ignoring malparsed backtrace: 2022-09-04T14:35:43.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:43.718 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:43.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:35:43.742 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:43.742 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:35:43.743 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:43.767 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:43.767 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:43.768 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:43.893+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 , 2022-09-04T14:35:43.959 INFO:teuthology.orchestra.run.smithi155.stdout:{ , 2022-09-04T14:35:43.960 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], Ignoring malparsed backtrace: 2022-09-04T14:35:44.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:44.255 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:44.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:35:44.279 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:80 3.5s0 3:a9d45bed:::smithi16339483-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:35:44.279 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.765917+0000", , 2022-09-04T14:35:44.279 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 91.464157826999994, Ignoring malparsed backtrace: 2022-09-04T14:35:44.303 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:81 3.5s0 3:a9d45bed:::smithi16339483-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:35:44.303 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.765934+0000", , 2022-09-04T14:35:44.303 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 91.464140592999996, Ignoring malparsed backtrace: 2022-09-04T14:35:44.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:35:44.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:44.330 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:44.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9360000000000001e-06 , 2022-09-04T14:35:44.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:44.359 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:44.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2220000000000001e-06 , 2022-09-04T14:35:44.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:44.383 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:44.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.669e-05 , 2022-09-04T14:35:44.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:44.412 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:44.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4320000000000006e-05 , 2022-09-04T14:35:44.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:44.439 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:44.464 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:44.465 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:44.465 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:44.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784610+0000", , 2022-09-04T14:35:44.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0583999999999997e-05 , 2022-09-04T14:35:44.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:44.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.785111+0000", , 2022-09-04T14:35:44.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.861e-06 , 2022-09-04T14:35:44.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:44.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0083998100000000006, , 2022-09-04T14:35:44.540 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:44.540 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:35:44.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011064098, , 2022-09-04T14:35:44.568 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:44.568 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:35:44.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53432486700000004, , 2022-09-04T14:35:44.595 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:44.596 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:35:44.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:35:44.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.174801+0000", , 2022-09-04T14:35:44.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001100932 Ignoring malparsed backtrace: 2022-09-04T14:35:44.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:44.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:21.266460+0000", , 2022-09-04T14:35:44.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2180000000000005e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:44.279+0000 7f814e78e700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset e=[]:{e=[e,c,b,9]} for object 3:0dcdd893:::smithi16339483-22:8 , 2022-09-04T14:35:44.383+0000 7faa2768e700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{2b=[2b,27,23,14],33=[33,2f],43=[43,42]} for object 3:70fb2152:::smithi16339483-29:9 , 2022-09-04T14:35:44.384+0000 7f4035433700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 Ignoring malparsed backtrace: 2022-09-04T14:35:45.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:35:45.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.752830+0000", , 2022-09-04T14:35:45.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5714000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:45.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:35:45.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.753438+0000", , 2022-09-04T14:35:45.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6011999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:45.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:35:45.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.005834+0000", , 2022-09-04T14:35:45.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031201369999999998 Ignoring malparsed backtrace: 2022-09-04T14:35:45.209 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:45.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:35:45.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801792+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:45.237 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:45.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:45.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.815461+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:45.260 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:45.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:45.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816949+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:45.283 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:45.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:35:45.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.913632+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:45.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:45.306 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:35:45.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:45.328 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:35:45.328 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:35:45.329 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:45.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:35:45.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:45.351 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:45.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001977253 , 2022-09-04T14:35:45.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:45.373 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:45.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027655959999999999 , 2022-09-04T14:35:45.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:45.396 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:45.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5962999999999999e-05 , 2022-09-04T14:35:45.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:35:45.418 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:45.441 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:35:45.441 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:35:45.441 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:35:45.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119210+0000", , 2022-09-04T14:35:45.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3195000000000001e-05 , 2022-09-04T14:35:45.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:45.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.176042+0000", , 2022-09-04T14:35:45.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0326000000000001e-05 , 2022-09-04T14:35:45.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:47.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , 2022-09-04T14:35:47.209+0000 7ffa4ea67700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 38946) UID: 0 , 2022-09-04T14:35:47.274 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:35:48.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:35:48.096 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", , 2022-09-04T14:35:48.097 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:35:48.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.724116+0000", , 2022-09-04T14:35:48.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:35:48.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:48.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.729587+0000", , 2022-09-04T14:35:48.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001060542 , 2022-09-04T14:35:48.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:48.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.729853+0000", , 2022-09-04T14:35:48.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013047040000000001 , 2022-09-04T14:35:48.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:48.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.753225+0000", , 2022-09-04T14:35:48.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000463053 , 2022-09-04T14:35:48.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:48.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.761662+0000", , 2022-09-04T14:35:48.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082105270000000004 , 2022-09-04T14:35:48.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:48.240 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:35:48.240 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 , 2022-09-04T14:35:48.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:48.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:35:48.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.860979+0000", , 2022-09-04T14:35:48.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:48.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:48.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.782165+0000", , 2022-09-04T14:35:48.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001891453 Ignoring malparsed backtrace: 2022-09-04T14:35:48.310 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:48.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:35:48.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910370+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:48.335 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:48.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:48.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916341+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:48.361 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:35:48.361 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:35:48.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:48.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:35:48.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", , 2022-09-04T14:35:48.386 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:35:48.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:48.408 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:48.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:48.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:48.432 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:48.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:48.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.89e-06 , 2022-09-04T14:35:48.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:48.455 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:48.977 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:48.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:35:48.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:13.004160+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:35:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.774863+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:35:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:49.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:49.083 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:49.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:35:49.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:49.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:49.110 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:49.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4331000000000001e-05 , 2022-09-04T14:35:49.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:49.135 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:49.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090612939999999993 , 2022-09-04T14:35:49.161 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:35:49.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:35:49.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.085860+0000", , 2022-09-04T14:35:49.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967296 , 2022-09-04T14:35:49.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:49.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.090490+0000", , 2022-09-04T14:35:49.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0600000000000002e-06 , 2022-09-04T14:35:49.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:49.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.144187+0000", , 2022-09-04T14:35:49.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017012959000000001 , 2022-09-04T14:35:49.240 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:49.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:49.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.130905+0000", , 2022-09-04T14:35:49.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967296 Ignoring malparsed backtrace: 2022-09-04T14:35:49.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:35:49.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.133811+0000", , 2022-09-04T14:35:49.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1157e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:49.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:49.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:10.129878+0000", , 2022-09-04T14:35:49.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018831300000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:49.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:49.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:16.182065+0000", , 2022-09-04T14:35:49.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098966420000000006 Ignoring malparsed backtrace: 2022-09-04T14:35:49.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub , 2022-09-04T14:35:49.758+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:35:49.820 INFO:teuthology.orchestra.run.smithi038.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:35:49.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:49.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783973+0000", , 2022-09-04T14:35:49.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:35:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:35:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833188+0000", , 2022-09-04T14:35:49.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8930000000000005e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:49.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:49.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.840927+0000", , 2022-09-04T14:35:49.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067311599999999999 Ignoring malparsed backtrace: 2022-09-04T14:35:49.925 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:49.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:35:49.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834326+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:35:49.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.777357+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:49.975 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:49.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:35:49.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.786750+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:50.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:50.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:50.024 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:50.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:50.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:50.057 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:35:50.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:50.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:50.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:50.085 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:50.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9427999999999998e-05 , 2022-09-04T14:35:50.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:50.107 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:50.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8788e-05 , 2022-09-04T14:35:50.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:35:50.130 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:50.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126488+0000", , 2022-09-04T14:35:50.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:35:50.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.130065+0000", , 2022-09-04T14:35:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3062999999999997e-05 , 2022-09-04T14:35:50.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:17.177499+0000", , 2022-09-04T14:35:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011418820000000001 , 2022-09-04T14:35:50.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:51.716+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:35:51.816+0000 7efdf07f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38715) UID: 0 , 2022-09-04T14:35:51.838 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:35:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 , 2022-09-04T14:35:53.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring , 2022-09-04T14:35:53.633+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:53.807 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012067081, , 2022-09-04T14:35:53.807 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:35:53.807 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:35:53.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:53.833 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.841104+0000", , 2022-09-04T14:35:53.833 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:53.858 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:35:53.858 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.841116+0000", , 2022-09-04T14:35:53.858 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 Ignoring malparsed backtrace: 2022-09-04T14:35:53.884 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:53.884 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843329+0000", , 2022-09-04T14:35:53.884 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1840000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:53.909 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", , 2022-09-04T14:35:53.909 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.695790+0000", , 2022-09-04T14:35:53.909 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01268525 Ignoring malparsed backtrace: 2022-09-04T14:35:53.934 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:35:53.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:53.935 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:53.960 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:35:53.960 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:35:53.960 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 635 Ignoring malparsed backtrace: 2022-09-04T14:35:53.985 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:53.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:35:53.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.073708+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.010 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:54.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.069525+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.035 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:54.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.062451+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.060 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:54.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:54.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.086 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:54.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0701e-05 , 2022-09-04T14:35:54.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.111 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:54.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.009999282 , 2022-09-04T14:35:54.135 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:35:54.136 INFO:teuthology.orchestra.run.smithi163.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:35:54.160 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:35:54.160 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", , 2022-09-04T14:35:54.160 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:35:54.583 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 101.850921687, , 2022-09-04T14:35:54.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013886004, , 2022-09-04T14:35:54.584 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:54.609 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 101.850894805, , 2022-09-04T14:35:54.610 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023000298999999998, , 2022-09-04T14:35:54.610 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:35:54.652 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:35:54.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.754545+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.684 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:54.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776574+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.711 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:35:54.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776726+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.740 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:54.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.780473+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.765 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:35:54.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.790690+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.789 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.789 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:54.789 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.795098+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.813 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.813 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:54.813 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.807732+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:35:54.836 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:35:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.861 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:35:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.886 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:54.887 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:35:54.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.912 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010484019999999999 , 2022-09-04T14:35:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.936 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:54.961 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00049445999999999995 , 2022-09-04T14:35:54.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.962 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:54.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012534626 , 2022-09-04T14:35:54.986 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:54.987 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:55.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01274632 , 2022-09-04T14:35:55.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:55.011 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:55.036 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:35:55.036 INFO:teuthology.orchestra.run.smithi163.stdout:} , 2022-09-04T14:35:55.037+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:55.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.781001+0000", , 2022-09-04T14:35:55.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029897099999999998 , 2022-09-04T14:35:55.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:55.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.794231+0000", , 2022-09-04T14:35:55.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001060444 , 2022-09-04T14:35:55.524 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:55.549 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:35:55.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.832831+0000", , 2022-09-04T14:35:55.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 Ignoring malparsed backtrace: 2022-09-04T14:35:55.573 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:35:55.573 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833113+0000", , 2022-09-04T14:35:55.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7189999999999996e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:55.597 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:55.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843463+0000", , 2022-09-04T14:35:55.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0095069549999999992 Ignoring malparsed backtrace: 2022-09-04T14:35:55.621 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:55.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:35:55.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.835167+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:55.645 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:55.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:35:55.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.838697+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:55.670 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:55.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:35:55.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.097460+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:55.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:55.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:55.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:55.719 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:55.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:55.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:55.743 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:55.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:35:55.768 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs5 99/52 ECSubWrite(tid=118, reqid=client.4557.0:915, at_version=99'70, trim_to=0'0, roll_forward_to=89'67))", , 2022-09-04T14:35:55.769 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:16.172880+0000", , 2022-09-04T14:35:55.769 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.305032763, Ignoring malparsed backtrace: 2022-09-04T14:35:55.793 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1526999999999999e-05 , 2022-09-04T14:35:55.793 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:55.794 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:55.817 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000139383 , 2022-09-04T14:35:55.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:55.818 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:55.841 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:55.841 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.bs4 105/33 ECSubWrite(tid=274, reqid=client.4557.0:940, at_version=105'52, trim_to=0'0, roll_forward_to=92'50))", , 2022-09-04T14:35:55.842 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:22.192220+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:56.167 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} , 2022-09-04T14:35:56.176 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops , 2022-09-04T14:35:56.187 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} Ignoring malparsed backtrace: 2022-09-04T14:35:56.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:56.329 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:56.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:35:56.357 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:56.358 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:56.358 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:35:56.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:56.384 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:56.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:35:56.409 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:56.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:56.410 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1270000000000002e-06 , 2022-09-04T14:35:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:56.434 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3733999999999997e-05 , 2022-09-04T14:35:56.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:35:56.463 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.732801+0000", , 2022-09-04T14:35:56.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:35:56.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:56.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:41.957558+0000", , 2022-09-04T14:35:56.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4290000000000004e-06 , 2022-09-04T14:35:56.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:56.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:41.960336+0000", , 2022-09-04T14:35:56.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000181447 , 2022-09-04T14:35:56.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:56.567 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:56.567 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:45.967225+0000", , 2022-09-04T14:35:56.567 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:35:56.591 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:56.592 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.089308+0000", , 2022-09-04T14:35:56.592 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4030000000000001e-06 Ignoring malparsed backtrace: 2022-09-04T14:35:56.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:56.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.171593+0000", , 2022-09-04T14:35:56.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8084999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:35:56.644 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:56.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:35:56.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.172695+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:56.676 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:56.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:35:56.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175529+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:56.700 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:35:56.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:35:56.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.272688+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:35:57.868 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-13, ['dump_ops_in_flight'] , 2022-09-04T14:35:57.918+0000 7f4052883700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38941) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:35:59.065 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:59.066 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.781832+0000", , 2022-09-04T14:35:59.066 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00011127400000000001 Ignoring malparsed backtrace: 2022-09-04T14:35:59.090 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:35:59.090 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.090 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:59.114 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:59.114 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:35:59.114 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.784593+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:59.138 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:59.138 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:35:59.138 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.784948+0000", Ignoring malparsed backtrace: 2022-09-04T14:35:59.162 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:35:59.163 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:35:59.163 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:59.186 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.186 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:59.187 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:35:59.210 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.210 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:59.210 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:35:59.233 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:35:59.234 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:35:59.234 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:59.257 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967296 , 2022-09-04T14:35:59.258 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.258 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:59.285 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.0456e-05 , 2022-09-04T14:35:59.285 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.285 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:59.307 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.004231299 , 2022-09-04T14:35:59.308 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:35:59.308 INFO:teuthology.orchestra.run.smithi077.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:35:59.330 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.072753+0000", , 2022-09-04T14:35:59.330 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967296 , 2022-09-04T14:35:59.330 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:59.353 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:09.132053+0000", , 2022-09-04T14:35:59.353 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.3780000000000001e-06 , 2022-09-04T14:35:59.353 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:35:59.375 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:21.269645+0000", , 2022-09-04T14:35:59.376 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.003712117 , 2022-09-04T14:35:59.376 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:35:59.398 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:35:59.398 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194301+0000", , 2022-09-04T14:35:59.398 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:35:59.757+0000 7efdcd395700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2f=[]:{2f=[2f]} for object 3:b771c3b4:::smithi16339483-5:9 , 2022-09-04T14:35:59.760+0000 7fc7f1867700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset c=[]:{c=[c,b,9]} for object 3:da2cfe71:::smithi16339483-46:7 , 2022-09-04T14:35:59.760+0000 7fc7eb85b700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 17=[]:{9=[9],c=[c,b],17=[17,14,12,e]} for object 3:eaf642d1:::smithi16339483-45:5 Ignoring malparsed backtrace: 2022-09-04T14:35:59.837 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.838 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:59.838 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:35:59.861 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.861 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:59.862 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:35:59.885 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.885 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:35:59.885 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:35:59.908 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:35:59.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.908 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:59.931 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.8808999999999999e-05 , 2022-09-04T14:35:59.931 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.932 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:59.955 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.0513000000000002e-05 , 2022-09-04T14:35:59.955 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:35:59.955 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:35:59.978 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684486+0000", , 2022-09-04T14:35:59.978 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:35:59.978 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:00.001 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684830+0000", , 2022-09-04T14:36:00.001 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00023506300000000001 , 2022-09-04T14:36:00.001 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:00.024 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:35.277669+0000", , 2022-09-04T14:36:00.025 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.5621999999999999e-05 , 2022-09-04T14:36:00.025 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:00.048 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:36:00.048 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971378+0000", , 2022-09-04T14:36:00.048 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:00.071 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:36:00.072 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973239+0000", , 2022-09-04T14:36:00.072 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.7998000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:00.094 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:00.095 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:46.967096+0000", , 2022-09-04T14:36:00.095 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.0358999999999994e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:00.118 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.118 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:36:00.118 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.074301+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:00.141 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.141 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:36:00.141 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.103537+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:00.168 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.168 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:00.168 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.105722+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:00.593 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0017951079999999999 , 2022-09-04T14:36:00.593 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:00.593 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:00.617 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.788242+0000", , 2022-09-04T14:36:00.617 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:36:00.617 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:00.641 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.815599+0000", , 2022-09-04T14:36:00.641 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.5290000000000003e-06 , 2022-09-04T14:36:00.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:00.667 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.817020+0000", , 2022-09-04T14:36:00.667 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.5417000000000002e-05 , 2022-09-04T14:36:00.667 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:00.695 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:36:00.695 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.832785+0000", , 2022-09-04T14:36:00.696 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:00.724 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:00.724 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833000+0000", , 2022-09-04T14:36:00.725 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.1270000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:00.748 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:36:00.748 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833445+0000", , 2022-09-04T14:36:00.749 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.2311999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:00.778 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.779 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:36:00.779 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971409+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:00.805 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.805 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:00.805 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973181+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:00.834 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.835 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:36:00.835 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.071269+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:00.859 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:36:00.859 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.859 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:36:00.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:00.886 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.886 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:00.911 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:00.911 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:00.911 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:36:00.938 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:00.938 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:36:00.938 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:00.968 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.6288e-05 , 2022-09-04T14:36:00.968 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:00.968 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:01.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.443 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:01.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:36:01.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.467 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:01.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:01.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.491 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:01.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:36:01.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.515 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:01.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:01.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.539 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:01.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:01.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7230000000000004e-06 , 2022-09-04T14:36:01.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.563 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:01.586 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 336 , 2022-09-04T14:36:01.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.587 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:01.610 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 337 , 2022-09-04T14:36:01.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.611 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:01.634 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 338 , 2022-09-04T14:36:01.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.634 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:01.658 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 339 , 2022-09-04T14:36:01.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:01.658 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.796801+0000", , 2022-09-04T14:36:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2310000000000004e-06 , 2022-09-04T14:36:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.067522+0000", , 2022-09-04T14:36:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9200000000000004e-06 , 2022-09-04T14:36:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.067566+0000", , 2022-09-04T14:36:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4079999999999998e-06 , 2022-09-04T14:36:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069448+0000", , 2022-09-04T14:36:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7355000000000002e-05 , 2022-09-04T14:36:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.074421+0000", , 2022-09-04T14:36:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6999000000000001e-05 , 2022-09-04T14:36:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:01.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:59.058079+0000", , 2022-09-04T14:36:01.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0055491899999999999 , 2022-09-04T14:36:01.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:01.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.174253+0000", , 2022-09-04T14:36:01.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001327681 , 2022-09-04T14:36:01.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:01.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.174811+0000", , 2022-09-04T14:36:01.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018836160000000001 , 2022-09-04T14:36:01.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:03.266+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 , 2022-09-04T14:36:03.285+0000 7f403142b700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 , 2022-09-04T14:36:03.324 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:36:05.185 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ , 2022-09-04T14:36:05.186 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:05.186 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.as5 42/33 ECSubWrite(tid=47, reqid=client.4557.0:198, at_version=42'19, trim_to=0'0, roll_forward_to=41'18))", Ignoring malparsed backtrace: 2022-09-04T14:36:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:36:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.800340+0000", , 2022-09-04T14:36:05.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016438299999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:05.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:19.064328+0000", , 2022-09-04T14:36:05.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017419700000000001 Ignoring malparsed backtrace: 2022-09-04T14:36:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:05.271 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:36:05.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.782624+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683977+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:36:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:36:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:36:05.435 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:05.435 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:36:05.435 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:05.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 , 2022-09-04T14:36:05.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:05.460 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:05.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0480000000000004e-06 , 2022-09-04T14:36:05.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:05.488 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:05.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0063498929999999997 , 2022-09-04T14:36:05.512 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:05.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:36:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.172694+0000", , 2022-09-04T14:36:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:05.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:05.560 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175600+0000", , 2022-09-04T14:36:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0538e-05 , 2022-09-04T14:36:05.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.280482+0000", , 2022-09-04T14:36:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0077751130000000002 , 2022-09-04T14:36:05.589 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:36:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.842585+0000", , 2022-09-04T14:36:06.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011586185000000001 Ignoring malparsed backtrace: 2022-09-04T14:36:06.061 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:06.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:36:06.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.842576+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:36:06.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:24.311622+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:06.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:06.112 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:06.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:36:06.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:06.135 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:06.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:36:06.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:06.157 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:06.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:589 3.6s0 3:644c6dd7:::smithi16339483-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", , 2022-09-04T14:36:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:45.962857+0000", , 2022-09-04T14:36:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 80.052372676000005, Ignoring malparsed backtrace: 2022-09-04T14:36:06.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.90312430600000004 , 2022-09-04T14:36:06.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:06.203 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:06.225 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:06.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:06.226 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:06.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077499+0000", , 2022-09-04T14:36:06.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6459999999999996e-05 , 2022-09-04T14:36:06.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.078198+0000", , 2022-09-04T14:36:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7079000000000002e-05 , 2022-09-04T14:36:06.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:06.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01289934, , 2022-09-04T14:36:06.294 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:06.294 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:36:06.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:36:06.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.131667+0000", , 2022-09-04T14:36:06.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5179000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:06.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:36:06.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.188522+0000", , 2022-09-04T14:36:06.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011697159999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:06.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:36:06.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.188713+0000", , 2022-09-04T14:36:06.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013040619999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:06.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:36:06.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.722774+0000", , 2022-09-04T14:36:06.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9359999999999994e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:06.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:36:06.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.729510+0000", , 2022-09-04T14:36:06.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041688439999999997 Ignoring malparsed backtrace: 2022-09-04T14:36:06.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:06.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.737938+0000", , 2022-09-04T14:36:06.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2009999999999996e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:36:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.752064+0000", , 2022-09-04T14:36:06.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001010742 Ignoring malparsed backtrace: 2022-09-04T14:36:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:36:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.761399+0000", , 2022-09-04T14:36:06.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8588e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:36:06.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.773199+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:07.000 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:07.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:07.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.830570+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:07.023 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:07.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", , 2022-09-04T14:36:07.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.861963+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:07.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:07.047 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:07.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:36:07.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:07.071 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:07.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:07.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:07.094 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:07.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:36:07.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:07.118 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:07.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0020000000000001e-05 , 2022-09-04T14:36:07.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:07.141 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:07.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0127999999999999e-05 , 2022-09-04T14:36:07.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:07.165 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:07.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.103146+0000", , 2022-09-04T14:36:07.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:36:07.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:07.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:15.168114+0000", , 2022-09-04T14:36:07.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9753000000000006e-05 , 2022-09-04T14:36:07.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: Error ENXIO: problem getting command descriptions from osd.13 , osd.14: osd_enable_op_tracker = 'false' , osd.15: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:36:10.581+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 , 2022-09-04T14:36:10.603 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} , 2022-09-04T14:36:10.606 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:36:11.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749820+0000", , 2022-09-04T14:36:11.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5460000000000002e-06 , 2022-09-04T14:36:11.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:11.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.749839+0000", , 2022-09-04T14:36:11.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1330000000000001e-06 , 2022-09-04T14:36:11.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:11.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.002527+0000", , 2022-09-04T14:36:11.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0981000000000003e-05 , 2022-09-04T14:36:11.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:11.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.006152+0000", , 2022-09-04T14:36:11.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4447000000000001e-05 , 2022-09-04T14:36:11.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:11.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:36:11.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.815258+0000", , 2022-09-04T14:36:11.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:11.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:36:11.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816639+0000", , 2022-09-04T14:36:11.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0136999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:11.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:11.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.843805+0000", , 2022-09-04T14:36:11.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000238488 Ignoring malparsed backtrace: 2022-09-04T14:36:11.543 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:36:11.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:11.543 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", , 2022-09-04T14:36:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949168+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:36:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:36:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:36:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:36:11.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:11.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:11.702 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:11.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5300000000000002e-06 , 2022-09-04T14:36:11.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:11.725 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:12.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:12.200 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:15.951726+0000", , 2022-09-04T14:36:12.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4790000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:12.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:12.223 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.784999+0000", , 2022-09-04T14:36:12.223 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021769000000000001 Ignoring malparsed backtrace: 2022-09-04T14:36:12.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:36:12.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.845208+0000", , 2022-09-04T14:36:12.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001172368 Ignoring malparsed backtrace: 2022-09-04T14:36:12.270 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:36:12.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843056+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:12.293 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:36:12.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680008+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:12.316 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:36:12.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.681161+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:12.339 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.339 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:36:12.339 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.682957+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:12.362 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.362 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:12.362 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.083506+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:12.385 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.385 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:36:12.385 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.085327+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:12.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:12.408 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.408 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:36:12.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:12.431 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting_for_map not empty", Ignoring malparsed backtrace: 2022-09-04T14:36:12.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7239999999999998e-06 , 2022-09-04T14:36:12.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:12.455 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:12.477 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:12.477 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs1 99/52 ECSubWrite(tid=118, reqid=client.4557.0:915, at_version=99'70, trim_to=0'0, roll_forward_to=89'67))", , 2022-09-04T14:36:12.478 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:16.171689+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:12.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173190+0000", , 2022-09-04T14:36:12.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6869e-05 , 2022-09-04T14:36:12.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:12.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.247583+0000", , 2022-09-04T14:36:12.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000117174 , 2022-09-04T14:36:12.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:12.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.264100+0000", , 2022-09-04T14:36:12.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5784000000000003e-05 , 2022-09-04T14:36:12.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:13.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.018 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:13.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2022-09-04T14:36:13.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.044 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:13.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2022-09-04T14:36:13.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.071 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:13.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:36:13.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.094 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:13.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2022-09-04T14:36:13.120 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:36:13.120 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:36:13.120 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:36:13.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:13.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.144 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:13.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:13.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.170 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:13.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5699999999999993e-06 , 2022-09-04T14:36:13.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.193 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:13.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0609999999999998e-06 , 2022-09-04T14:36:13.215 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.215 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:13.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001716724 , 2022-09-04T14:36:13.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.238 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:13.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033903299999999998 , 2022-09-04T14:36:13.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.261 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:13.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.792300+0000", , 2022-09-04T14:36:13.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:13.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:13.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.912937+0000", , 2022-09-04T14:36:13.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2362e-05 , 2022-09-04T14:36:13.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:13.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:13.330 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:13.330 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:353 3.9s0 3:96c14a12:::smithi16339483-31:head [append 0~0] snapc e=[e,c,b,9,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", Ignoring malparsed backtrace: 2022-09-04T14:36:13.354 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:36:13.354 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 856 , 2022-09-04T14:36:13.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:13.377 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:36:13.378 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 857 , 2022-09-04T14:36:13.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:13.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:36:13.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.263553+0000", , 2022-09-04T14:36:13.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 Ignoring malparsed backtrace: 2022-09-04T14:36:13.912 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} , osd.0: osd_enable_op_tracker = 'false' , osd.1: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:36:16.044+0000 7fc80fcb9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 39177) UID: 0 , 2022-09-04T14:36:16.069+0000 7faa2b696700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{2b=[2b,27,23,14],33=[33,2f],43=[43,42]} for object 3:70fb2152:::smithi16339483-29:9 , 2022-09-04T14:36:16.070+0000 7f4e01c5f700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{c=[c],15=[15,14,12,11,e],1e=[1e,1b,17],2b=[27,23],43=[42]} for object 3:38890410:::smithi16339483-19:9 Ignoring malparsed backtrace: 2022-09-04T14:36:16.845 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00011127400000000001 , 2022-09-04T14:36:16.846 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:16.846 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:16.870 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:16.871 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.2s5 42/41 ECSubWrite(tid=12, reqid=client.4557.0:204, at_version=42'9, trim_to=0'0, roll_forward_to=0'0))", , 2022-09-04T14:36:16.871 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:18.783649+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:16.894 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.784593+0000", , 2022-09-04T14:36:16.894 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.8321999999999998e-05 , 2022-09-04T14:36:16.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:16.917 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.784948+0000", , 2022-09-04T14:36:16.917 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.4426999999999997e-05 , 2022-09-04T14:36:16.918 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:16.942 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:16.942 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:16.942 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.9s1 42/33 ECSubWrite(tid=30, reqid=client.4557.0:206, at_version=42'18, trim_to=0'0, roll_forward_to=37'16))", Ignoring malparsed backtrace: 2022-09-04T14:36:16.966 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:36:16.967 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.790605+0000", , 2022-09-04T14:36:16.967 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000244546 Ignoring malparsed backtrace: 2022-09-04T14:36:16.992 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:16.992 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.807804+0000", , 2022-09-04T14:36:16.992 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00014063699999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:17.015 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:17.015 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.016 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:17.039 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.039 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:36:17.039 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.683938+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:17.062 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.062 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:17.063 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:34.350011+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:17.086 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:36:17.086 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:17.086 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:17.110 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.110 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.110 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:36:17.133 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.133 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.133 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:36:17.157 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:17.157 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:36:17.157 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:17.180 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.999999 , 2022-09-04T14:36:17.181 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.181 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:17.684 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 124.90589931300001, , 2022-09-04T14:36:17.684 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.036370176999999997, , 2022-09-04T14:36:17.684 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:36:17.707 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.707 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:36:17.707 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.814392+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:17.729 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.729 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:17.729 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831851+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:17.752 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:23.834389+0000", , 2022-09-04T14:36:17.752 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 113.845348058, , 2022-09-04T14:36:17.752 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.01066607, Ignoring malparsed backtrace: 2022-09-04T14:36:17.774 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.774 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.775 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:36:17.796 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.797 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.797 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:36:17.818 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.cs6 47/33 ECSubWrite(tid=21, reqid=client.4557.0:312, at_version=47'15, trim_to=0'0, roll_forward_to=45'11))", , 2022-09-04T14:36:17.819 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:23.843585+0000", , 2022-09-04T14:36:17.819 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 113.836152365, Ignoring malparsed backtrace: 2022-09-04T14:36:17.841 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.9203000000000001e-05 , 2022-09-04T14:36:17.841 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.841 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:17.863 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.2948999999999997e-05 , 2022-09-04T14:36:17.863 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.864 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:17.885 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.885 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.1s3 63/52 ECSubWrite(tid=158, reqid=client.4557.0:486, at_version=63'30, trim_to=0'0, roll_forward_to=56'28))", , 2022-09-04T14:36:17.886 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:39.968393+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:17.908 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971408+0000", , 2022-09-04T14:36:17.908 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.0837000000000003e-05 , 2022-09-04T14:36:17.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973363+0000", , 2022-09-04T14:36:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00011252999999999999 , 2022-09-04T14:36:17.930 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:17.952 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:17.953 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:17.953 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.1s3 81/52 ECSubWrite(tid=203, reqid=client.4557.0:689, at_version=81'36, trim_to=0'0, roll_forward_to=66'34))", Ignoring malparsed backtrace: 2022-09-04T14:36:17.976 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:36:17.976 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.075499+0000", , 2022-09-04T14:36:17.976 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.001198855 Ignoring malparsed backtrace: 2022-09-04T14:36:17.999 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:17.999 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.103683+0000", , 2022-09-04T14:36:17.999 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00013278000000000001 Ignoring malparsed backtrace: 2022-09-04T14:36:18.022 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:18.023 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:18.023 INFO:teuthology.orchestra.run.smithi077.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:36:18.481 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:18.481 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:36:18.482 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.793258+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:18.504 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:36:18.527 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:18.528 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:18.528 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:36:18.550 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:18.551 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:18.551 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:18.573 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:36:18.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:18.573 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:18.596 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000204868 , 2022-09-04T14:36:18.596 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:18.596 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:18.619 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00021678499999999999 , 2022-09-04T14:36:18.619 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:18.619 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:18.641 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971409+0000", , 2022-09-04T14:36:18.642 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:36:18.642 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:18.664 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973360+0000", , 2022-09-04T14:36:18.664 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00017937399999999999 , 2022-09-04T14:36:18.665 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:18.687 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.071289+0000", , 2022-09-04T14:36:18.687 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.0369000000000001e-05 , 2022-09-04T14:36:18.687 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:18.710 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:36:18.710 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.074404+0000", , 2022-09-04T14:36:18.710 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:18.734 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:36:18.734 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.081921+0000", , 2022-09-04T14:36:18.734 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00011809 Ignoring malparsed backtrace: 2022-09-04T14:36:18.757 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:18.758 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.084173+0000", , 2022-09-04T14:36:18.758 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2906e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:18.781 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:18.781 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:36:18.781 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192194+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:18.804 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:18.805 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:36:18.805 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194411+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:19.244 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:19.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:36:19.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.826410+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:19.269 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:19.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:36:19.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828877+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:19.303 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:19.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:36:19.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.829697+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:19.330 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:19.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:36:19.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.842141+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:19.353 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:19.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:19.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.819730+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:19.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.376 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:19.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:36:19.399 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:336 3.fs0 3:f5330fd6:::smithi16339483-15:head [append 0~491520] snapc e=[e,c,b,9,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", , 2022-09-04T14:36:19.399 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:27.772026+0000", , 2022-09-04T14:36:19.400 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 111.46142070499999, Ignoring malparsed backtrace: 2022-09-04T14:36:19.422 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:337 3.fs0 3:f5330fd6:::smithi16339483-15:head [append 0~770048] snapc e=[e,c,b,9,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", , 2022-09-04T14:36:19.422 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:27.776402+0000", , 2022-09-04T14:36:19.423 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 111.4570439, Ignoring malparsed backtrace: 2022-09-04T14:36:19.445 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:338 3.fs0 3:f5330fd6:::smithi16339483-15:head [setxattr _header (56) in=7b] snapc e=[e,c,b,9,2] ondisk+write+known_if_redirected+supports_pool_eio e51)", , 2022-09-04T14:36:19.445 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:27.777099+0000", , 2022-09-04T14:36:19.446 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 111.45634748000001, Ignoring malparsed backtrace: 2022-09-04T14:36:19.469 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:339 3.fs0 3:f5330fd6:::smithi16339483-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", , 2022-09-04T14:36:19.469 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:27.777129+0000", , 2022-09-04T14:36:19.469 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 111.456317634, Ignoring malparsed backtrace: 2022-09-04T14:36:19.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:19.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.492 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:19.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:19.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.516 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:19.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:19.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.540 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:19.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012371100000000001 , 2022-09-04T14:36:19.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.564 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:19.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0433e-05 , 2022-09-04T14:36:19.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.587 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:19.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0203000000000004e-05 , 2022-09-04T14:36:19.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.611 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:19.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019313199999999999 , 2022-09-04T14:36:19.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.636 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:19.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8527e-05 , 2022-09-04T14:36:19.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:19.669 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:20.550+0000 7fc7eb85b700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 17=[]:{9=[9],c=[c,b],17=[17,14,12,e]} for object 3:eaf642d1:::smithi16339483-45:5 , 2022-09-04T14:36:20.585+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 , 2022-09-04T14:36:20.587+0000 7f403142b700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 Ignoring malparsed backtrace: 2022-09-04T14:36:22.845 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} , 2022-09-04T14:36:22.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , 2022-09-04T14:36:22.867 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} Ignoring malparsed backtrace: 2022-09-04T14:36:23.369 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:23.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:23.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.830899+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:23.400 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:23.842577+0000", , 2022-09-04T14:36:23.400 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 119.508553241, , 2022-09-04T14:36:23.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019339667000000001, Ignoring malparsed backtrace: 2022-09-04T14:36:23.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:23.431 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:23.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:36:23.458 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/33 ECSubWrite(tid=66, reqid=client.4557.0:353, at_version=53'48, trim_to=0'0, roll_forward_to=48'44))", , 2022-09-04T14:36:23.458 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:29.682567+0000", , 2022-09-04T14:36:23.458 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 113.66856411000001, Ignoring malparsed backtrace: 2022-09-04T14:36:23.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4161000000000001e-05 , 2022-09-04T14:36:23.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:23.486 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:23.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.747e-05 , 2022-09-04T14:36:23.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:23.515 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7058999999999998e-05 , 2022-09-04T14:36:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.965465+0000", , 2022-09-04T14:36:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:36:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:23.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.057784+0000", , 2022-09-04T14:36:23.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5031999999999999e-05 , 2022-09-04T14:36:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:36:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077396+0000", , 2022-09-04T14:36:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:36:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.078053+0000", , 2022-09-04T14:36:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:23.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:23.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126704+0000", , 2022-09-04T14:36:23.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121317 Ignoring malparsed backtrace: 2022-09-04T14:36:23.693 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:23.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:36:23.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.130692+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:23.733 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:23.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:23.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.187296+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:23.760 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:23.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:23.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.187323+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:24.294 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:24.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:36:24.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.722763+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:24.323 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:24.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:24.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.724130+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:24.350 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:24.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", , 2022-09-04T14:36:24.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.737436+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:24.373 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:24.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", , 2022-09-04T14:36:24.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.738376+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:24.396 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:24.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:36:24.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.753438+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:24.419 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:36:24.419 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:24.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:24.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:24.443 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:24.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:36:24.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:24.466 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:24.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:36:24.488 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:24.488 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:36:24.488 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:24.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.999999 , 2022-09-04T14:36:24.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:24.511 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:24.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0472000000000001e-05 , 2022-09-04T14:36:24.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:24.533 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:24.556 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 708 , 2022-09-04T14:36:24.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:24.557 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:24.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:59.046836+0000", , 2022-09-04T14:36:24.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:36:24.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:24.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.102868+0000", , 2022-09-04T14:36:24.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060840799999999995 , 2022-09-04T14:36:24.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:24.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.867676+0000", , 2022-09-04T14:36:24.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011243410000000001 , 2022-09-04T14:36:24.625 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:24.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:36:24.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:15.165583+0000", , 2022-09-04T14:36:24.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012320399999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:25.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:13.004277+0000", , 2022-09-04T14:36:25.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107704 , 2022-09-04T14:36:25.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:25.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:25.199 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:25.200 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.9s5 53/33 ECSubWrite(tid=65, reqid=client.4557.0:352, at_version=53'47, trim_to=0'0, roll_forward_to=48'44))", Ignoring malparsed backtrace: 2022-09-04T14:36:25.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:36:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682709+0000", , 2022-09-04T14:36:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8797999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:25.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:25.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.156542+0000", , 2022-09-04T14:36:25.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021441799999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:25.284 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:25.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:25.285 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:25.307 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:25.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:36:25.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910022+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:25.329 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:36:25.329 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:25.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:25.351 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:36:25.352 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:25.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:36:25.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:25.374 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:25.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:25.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:25.397 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:25.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:36:25.419 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:25.419 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:36:25.420 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:25.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6777000000000001e-05 , 2022-09-04T14:36:25.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:25.442 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:25.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6260000000000001e-06 , 2022-09-04T14:36:25.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:25.466 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:25.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5251000000000002e-05 , 2022-09-04T14:36:25.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:25.493 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:25.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:25.518 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:25.518 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:25.775+0000 7efdd139d700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2f=[]:{2f=[2f]} for object 3:b771c3b4:::smithi16339483-5:9 , 2022-09-04T14:36:25.839+0000 7ffa4ea67700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 38946) UID: 0 , 2022-09-04T14:36:25.893 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} Ignoring malparsed backtrace: 2022-09-04T14:36:28.104 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} , 2022-09-04T14:36:28.127 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} , 2022-09-04T14:36:28.149 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} Ignoring malparsed backtrace: 2022-09-04T14:36:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.722254+0000", , 2022-09-04T14:36:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 135.56420469599999, , 2022-09-04T14:36:28.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019991169999999999, Ignoring malparsed backtrace: 2022-09-04T14:36:28.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:28.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:36:28.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.366 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:28.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:28.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.397 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:28.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:28.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.420 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:28.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:28.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:28.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.445 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:28.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9410000000000001e-06 , 2022-09-04T14:36:28.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.471 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:28.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5980000000000001e-05 , 2022-09-04T14:36:28.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.503 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:28.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000120776 , 2022-09-04T14:36:28.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:28.530 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:28.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4021e-05 , 2022-09-04T14:36:28.556 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:36:28.556 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:36:28.583 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:36:28.584 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:36:28.584 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:36:28.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.085095+0000", , 2022-09-04T14:36:28.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:28.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:28.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.085147+0000", , 2022-09-04T14:36:28.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7049999999999999e-06 , 2022-09-04T14:36:28.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:28.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:36:28.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119394+0000", , 2022-09-04T14:36:28.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:28.689 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:28.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:36:28.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:15.158583+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:29.189 INFO:teuthology.orchestra.run.smithi155.stdout:{ , 2022-09-04T14:36:29.190 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, , 2022-09-04T14:36:29.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, Ignoring malparsed backtrace: 2022-09-04T14:36:29.213 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:36:29.213 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:36:29.213 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:29.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:29.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:29.235 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:29.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:29.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:29.258 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:29.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1722999999999999e-05 , 2022-09-04T14:36:29.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:29.280 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:29.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00095956199999999996 , 2022-09-04T14:36:29.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:29.303 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:29.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001069609 , 2022-09-04T14:36:29.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:29.325 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:29.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0180000000000004e-06 , 2022-09-04T14:36:29.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:29.348 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:29.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6855999999999999e-05 , 2022-09-04T14:36:29.370 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:36:29.370 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:36:29.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783673+0000", , 2022-09-04T14:36:29.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:36:29.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:29.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784665+0000", , 2022-09-04T14:36:29.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8630000000000008e-06 , 2022-09-04T14:36:29.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:29.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.794534+0000", , 2022-09-04T14:36:29.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093767590000000001 , 2022-09-04T14:36:29.437 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:29.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:36:29.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.830012+0000", , 2022-09-04T14:36:29.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:36:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.905044+0000", , 2022-09-04T14:36:29.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:36:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:56.032035+0000", , 2022-09-04T14:36:29.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:36:29.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:29.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.175821+0000", , 2022-09-04T14:36:29.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017596199999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:29.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:36:29.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:21.272124+0000", , 2022-09-04T14:36:29.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00323456 Ignoring malparsed backtrace: 2022-09-04T14:36:29.772+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 , 2022-09-04T14:36:29.873+0000 7f91016aa700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 39166) UID: 0 , 2022-09-04T14:36:29.880 INFO:teuthology.orchestra.run.smithi155.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:36:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013316200000000001 , 2022-09-04T14:36:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:30.039 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:36:30.061 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:36:30.062 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:36:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858431+0000", , 2022-09-04T14:36:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:30.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801860+0000", , 2022-09-04T14:36:30.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2700000000000004e-06 , 2022-09-04T14:36:30.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816160+0000", , 2022-09-04T14:36:30.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1546999999999998e-05 , 2022-09-04T14:36:30.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:30.151 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:36:30.151 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 312 , 2022-09-04T14:36:30.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:30.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:36:30.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.847606+0000", , 2022-09-04T14:36:30.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:36:30.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:36:30.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:24.340678+0000", , 2022-09-04T14:36:30.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0791999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:30.217 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:30.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:36:30.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949068+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:30.239 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:30.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:36:30.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.954237+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:30.262 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:30.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:30.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.954921+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:30.284 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:30.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:30.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.954922+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:30.306 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:36:30.306 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:30.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:36:30.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:30.330 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:30.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:30.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:30.352 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:30.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:36:30.036+0000 7fc7f1867700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset c=[]:{c=[c,b,9]} for object 3:da2cfe71:::smithi16339483-46:7 , 2022-09-04T14:36:30.036+0000 7efdcd395700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2f=[]:{2f=[2f]} for object 3:b771c3b4:::smithi16339483-5:9 , 2022-09-04T14:36:30.037+0000 7efdceb98700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 9=[]:{9=[9,8]} for object 3:55ca316a:::smithi16339483-9:5 Ignoring malparsed backtrace: 2022-09-04T14:36:30.855 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:30.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:30.855 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:30.879 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:36:30.880 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:36:30.880 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 308 Ignoring malparsed backtrace: 2022-09-04T14:36:30.922 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:30.923 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:36:30.923 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.841116+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:30.954 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:30.955 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:30.955 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843322+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:30.977 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683105+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:30.999 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:30.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:36:30.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.696055+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:31.021 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:51.073653+0000", , 2022-09-04T14:36:31.021 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 99.755461319000005, , 2022-09-04T14:36:31.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013582356, Ignoring malparsed backtrace: 2022-09-04T14:36:31.044 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:36:31.044 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:31.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:36:31.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:31.067 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:31.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:31.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:31.089 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:31.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:36:31.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:31.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:31.112 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:31.134 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 855 , 2022-09-04T14:36:31.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:31.134 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:31.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.171805+0000", , 2022-09-04T14:36:31.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4963999999999998e-05 , 2022-09-04T14:36:31.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:31.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173555+0000", , 2022-09-04T14:36:31.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4221999999999999e-05 , 2022-09-04T14:36:31.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:31.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:31.202 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:31.202 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:938 3.2s1 3:402e69e6:::smithi16339483-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", Ignoring malparsed backtrace: 2022-09-04T14:36:31.390 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:36:31.391 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:36:31.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:36:33.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops , 2022-09-04T14:36:33.745+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:36:33.750 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} Ignoring malparsed backtrace: 2022-09-04T14:36:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:34.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:36:34.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783357+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:34.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.787273+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:36:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:36:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:36:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6020000000000002e-06 , 2022-09-04T14:36:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4618999999999999e-05 , 2022-09-04T14:36:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:34.560 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910014+0000", , 2022-09-04T14:36:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:36:34.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916356+0000", , 2022-09-04T14:36:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0669999999999992e-06 , 2022-09-04T14:36:34.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954969+0000", , 2022-09-04T14:36:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7331999999999999e-05 , 2022-09-04T14:36:34.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:36:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.081604+0000", , 2022-09-04T14:36:34.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:34.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.084119+0000", , 2022-09-04T14:36:34.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.012e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:34.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:36:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.127067+0000", , 2022-09-04T14:36:34.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7269999999999995e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:34.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:36:34.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:17.175719+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:35.212 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:36:35.212 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:35.212 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:35.244 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.783118+0000", , 2022-09-04T14:36:35.244 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.6976999999999999e-05 , 2022-09-04T14:36:35.244 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:35.267 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.785479+0000", , 2022-09-04T14:36:35.267 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.001691138 , 2022-09-04T14:36:35.268 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:35.292 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0095973699999999992, , 2022-09-04T14:36:35.293 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:36:35.293 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:36:35.319 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:36:35.319 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.787957+0000", , 2022-09-04T14:36:35.319 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.2351999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:35.342 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:35.342 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.789958+0000", , 2022-09-04T14:36:35.343 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.3790000000000003e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:35.366 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 136.40183191, , 2022-09-04T14:36:35.366 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0041580940000000002, , 2022-09-04T14:36:35.366 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:36:35.391 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:35.392 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:36:35.392 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816529+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:35.423 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:35.423 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:35.424 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686383+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:35.450 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:33.682897+0000", , 2022-09-04T14:36:35.451 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 121.524378175, , 2022-09-04T14:36:35.451 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.66952835799999999, Ignoring malparsed backtrace: 2022-09-04T14:36:35.474 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:35.474 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:35.474 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:36:35.497 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:35.497 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:35.497 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:36:35.520 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.9s1 92/33 ECSubWrite(tid=133, reqid=client.4557.0:856, at_version=92'89, trim_to=0'0, roll_forward_to=90'88))", , 2022-09-04T14:36:35.520 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:35:09.130728+0000", , 2022-09-04T14:36:35.520 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 86.076546573000002, Ignoring malparsed backtrace: 2022-09-04T14:36:35.543 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0017864529999999999 , 2022-09-04T14:36:35.543 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:35.543 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:35.566 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000111193 , 2022-09-04T14:36:35.567 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:35.567 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:35.590 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:36:35.590 INFO:teuthology.orchestra.run.smithi077.stdout:} , 2022-09-04T14:36:35.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight Ignoring malparsed backtrace: 2022-09-04T14:36:36.051 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:12.809262+0000", , 2022-09-04T14:36:36.051 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 143.22027120199999, , 2022-09-04T14:36:36.051 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0056506730000000002, Ignoring malparsed backtrace: 2022-09-04T14:36:36.074 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:36.075 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:36.075 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:36:36.098 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:36.098 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:36.098 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:36:36.121 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.7s5 47/33 ECSubWrite(tid=87, reqid=client.4557.0:294, at_version=47'29, trim_to=0'0, roll_forward_to=47'25))", , 2022-09-04T14:36:36.121 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:23.835467+0000", , 2022-09-04T14:36:36.121 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 132.19406637399999, Ignoring malparsed backtrace: 2022-09-04T14:36:36.145 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.6461999999999998e-05 , 2022-09-04T14:36:36.145 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:36.145 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:36.168 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.9333999999999998e-05 , 2022-09-04T14:36:36.168 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:36.169 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:36.191 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:36.192 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.0s4 53/33 ECSubWrite(tid=69, reqid=client.4557.0:349, at_version=53'39, trim_to=0'0, roll_forward_to=51'37))", , 2022-09-04T14:36:36.192 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:29.684486+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:36.215 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684524+0000", , 2022-09-04T14:36:36.215 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.6359999999999994e-06 , 2022-09-04T14:36:36.215 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:36.238 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:35.277560+0000", , 2022-09-04T14:36:36.239 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000122862 , 2022-09-04T14:36:36.239 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:36.261 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:36.262 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:36.262 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.7s5 63/33 ECSubWrite(tid=160, reqid=client.4557.0:484, at_version=63'54, trim_to=0'0, roll_forward_to=61'52))", Ignoring malparsed backtrace: 2022-09-04T14:36:36.285 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:36:36.285 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973142+0000", , 2022-09-04T14:36:36.285 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2653999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:36.308 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:36.308 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:46.966942+0000", , 2022-09-04T14:36:36.309 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00011813 Ignoring malparsed backtrace: 2022-09-04T14:36:36.331 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:36.331 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:36.332 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:36.355 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:36.355 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:36:36.355 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.103441+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:36.378 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:36.378 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:36.378 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.105671+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:37.148 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:36:37.148 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:36:37.159 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops Ignoring malparsed backtrace: 2022-09-04T14:36:39.183 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:36:39.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:15.951711+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:39.210 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.210 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:36:39.210 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.784781+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:39.234 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:39.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.844036+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:39.257 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.843056+0000", , 2022-09-04T14:36:39.257 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 135.329611599, , 2022-09-04T14:36:39.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0098885899999999992, Ignoring malparsed backtrace: 2022-09-04T14:36:39.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:39.281 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:36:39.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:39.304 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:39.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:39.328 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:36:39.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:39.351 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.351 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:36:39.374 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:39.374 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:39.375 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:36:39.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:39.398 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:36:39.398 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:39.421 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.51e-06 , 2022-09-04T14:36:39.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:39.421 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:39.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.052259+0000", , 2022-09-04T14:36:39.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 , 2022-09-04T14:36:39.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:39.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.586075+0000", , 2022-09-04T14:36:39.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.466859939 , 2022-09-04T14:36:39.467 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:39.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:36:39.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173163+0000", , 2022-09-04T14:36:39.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:36:39.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:36:39.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.247466+0000", , 2022-09-04T14:36:39.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8293000000000005e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:39.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:36:39.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.264004+0000", , 2022-09-04T14:36:39.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016197135000000001 Ignoring malparsed backtrace: 2022-09-04T14:36:40.032 INFO:teuthology.orchestra.run.smithi163.stdout:{ , 2022-09-04T14:36:40.032 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, , 2022-09-04T14:36:40.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, Ignoring malparsed backtrace: 2022-09-04T14:36:40.057 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:40.058 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:36:40.058 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:40.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:40.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:40.083 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:40.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5359999999999998e-06 , 2022-09-04T14:36:40.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:40.108 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:40.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0600000000000002e-06 , 2022-09-04T14:36:40.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:40.134 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:40.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001369891 , 2022-09-04T14:36:40.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:40.159 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:40.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000372376 , 2022-09-04T14:36:40.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:40.183 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:40.207 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001373904 , 2022-09-04T14:36:40.208 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:40.208 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:40.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00089692700000000005 , 2022-09-04T14:36:40.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:40.233 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:40.257 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:36:40.257 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", , 2022-09-04T14:36:40.258 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:36:40.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.787757+0000", , 2022-09-04T14:36:40.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 , 2022-09-04T14:36:40.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:40.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.787783+0000", , 2022-09-04T14:36:40.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7699999999999998e-06 , 2022-09-04T14:36:40.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:40.400 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:36:40.400 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843561+0000", , 2022-09-04T14:36:40.400 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:40.423 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:40.423 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680164+0000", , 2022-09-04T14:36:40.423 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8729999999999998e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:40.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", , 2022-09-04T14:36:40.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.682023+0000", , 2022-09-04T14:36:40.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011351 Ignoring malparsed backtrace: 2022-09-04T14:36:40.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", , 2022-09-04T14:36:40.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.118239+0000", , 2022-09-04T14:36:40.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2970000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:40.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", , 2022-09-04T14:36:40.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.118444+0000", , 2022-09-04T14:36:40.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2354000000000006e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:40.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:36:40.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.276045+0000", , 2022-09-04T14:36:40.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011554200000000001 Ignoring malparsed backtrace: 2022-09-04T14:36:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8286000000000001e-05 , 2022-09-04T14:36:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107057 , 2022-09-04T14:36:41.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.065 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.832831+0000", , 2022-09-04T14:36:41.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:41.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:41.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833103+0000", , 2022-09-04T14:36:41.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8575999999999997e-05 , 2022-09-04T14:36:41.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:41.137 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:41.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:41.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833956+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:41.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.161 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:41.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:36:41.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.184 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:41.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:41.207 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.208 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:41.208 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:36:41.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:41.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.231 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5051000000000001e-05 , 2022-09-04T14:36:41.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.254 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2932000000000003e-05 , 2022-09-04T14:36:41.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.277 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0066918139999999999 , 2022-09-04T14:36:41.301 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:41.301 INFO:teuthology.orchestra.run.smithi163.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:36:41.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173190+0000", , 2022-09-04T14:36:41.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:41.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:41.347 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175555+0000", , 2022-09-04T14:36:41.347 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9379999999999998e-06 , 2022-09-04T14:36:41.347 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:41.370 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.183329+0000", , 2022-09-04T14:36:41.370 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0072104309999999998 , 2022-09-04T14:36:41.370 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:41.655 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], , 2022-09-04T14:36:41.655 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, , 2022-09-04T14:36:41.656 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:36:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:41.820 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:36:41.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.844 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:41.844 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:41.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.867 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:41.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:36:41.890 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:41.890 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:36:41.890 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.913 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014492800000000001 , 2022-09-04T14:36:41.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.914 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.936 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1099999999999994e-05 , 2022-09-04T14:36:41.936 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:41.937 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:41.959 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:36:41.960 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:41.960 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:41.982 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:41.957551+0000", , 2022-09-04T14:36:41.983 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4786999999999997e-05 , 2022-09-04T14:36:41.983 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:41.960154+0000", , 2022-09-04T14:36:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2790000000000004e-06 , 2022-09-04T14:36:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0063122919999999997, , 2022-09-04T14:36:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:36:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:36:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:36:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.089300+0000", , 2022-09-04T14:36:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2455e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.171545+0000", , 2022-09-04T14:36:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0981e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:42.098 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 85.622571394000005, , 2022-09-04T14:36:42.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018415279, , 2022-09-04T14:36:42.098 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:36:42.121 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:42.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:36:42.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175519+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:42.144 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:42.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:42.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.272239+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:43.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops , 2022-09-04T14:36:43.467+0000 7faf5177f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38712) UID: 0 , 2022-09-04T14:36:43.567 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:36:44.908 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:44.908 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:44.909 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:36:44.946 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:44.947 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:44.947 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:36:44.979 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:44.979 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:44.979 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:45.010 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:36:45.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:45.010 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:45.034 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.8315999999999999e-05 , 2022-09-04T14:36:45.035 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:45.035 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:45.070 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.4280000000000001e-05 , 2022-09-04T14:36:45.070 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:45.070 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:45.094 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843585+0000", , 2022-09-04T14:36:45.094 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:36:45.094 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:45.118 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684729+0000", , 2022-09-04T14:36:45.118 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000163079 , 2022-09-04T14:36:45.118 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:45.142 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684935+0000", , 2022-09-04T14:36:45.143 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.1464e-05 , 2022-09-04T14:36:45.143 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:45.166 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:36:45.167 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.968393+0000", , 2022-09-04T14:36:45.167 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:45.190 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:36:45.191 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971476+0000", , 2022-09-04T14:36:45.191 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.8877000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:45.215 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:45.215 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973422+0000", , 2022-09-04T14:36:45.216 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.3189e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:45.239 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:45.239 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:36:45.240 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.070130+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:45.263 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:45.263 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:36:45.263 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.075597+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:45.287 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:45.287 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:36:45.288 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.103747+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:45.354+0000 7efdceb98700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 9=[]:{9=[9,8]} for object 3:55ca316a:::smithi16339483-9:5 , 2022-09-04T14:36:45.354+0000 7faa2768e700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{2b=[2b,27,23,14],33=[33,2f],43=[43,42]} for object 3:70fb2152:::smithi16339483-29:9 , 2022-09-04T14:36:45.355+0000 7f4e01c5f700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{c=[c],15=[15,14,12,11,e],1e=[1e,1b,17],2b=[27,23],43=[42]} for object 3:38890410:::smithi16339483-19:9 Ignoring malparsed backtrace: 2022-09-04T14:36:45.843 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:45.843 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:45.843 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.bs6 42/33 ECSubWrite(tid=59, reqid=client.4557.0:197, at_version=42'11, trim_to=0'0, roll_forward_to=36'7))", Ignoring malparsed backtrace: 2022-09-04T14:36:45.875 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:36:45.875 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.788253+0000", , 2022-09-04T14:36:45.875 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2466000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:45.907 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:45.908 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.815848+0000", , 2022-09-04T14:36:45.908 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00010906199999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:45.937 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:45.937 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:45.937 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:45.962 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:45.962 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:36:45.962 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.832803+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:45.988 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:45.988 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:45.988 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833306+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:46.013 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:36:46.013 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:46.013 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:46.036 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:46.037 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:46.037 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:36:46.060 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:46.060 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:36:46.060 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:36:46.083 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:46.083 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:36:46.083 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:46.107 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:46.107 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:46.108 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:46.132 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.0178e-05 , 2022-09-04T14:36:46.132 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:36:46.132 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:46.164 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0053898799999999997 , 2022-09-04T14:36:46.164 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:36:46.164 INFO:teuthology.orchestra.run.smithi077.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:36:46.191 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192194+0000", , 2022-09-04T14:36:46.191 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:46.192 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:46.224 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194419+0000", , 2022-09-04T14:36:46.224 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.2429999999999999e-06 , 2022-09-04T14:36:46.225 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:36:46.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:46.815 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:46.815 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:36:46.837 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:36:46.837 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:36:46.837 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:46.859 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:36:46.859 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:36:46.860 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:46.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 , 2022-09-04T14:36:46.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:46.882 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:46.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.772026+0000", , 2022-09-04T14:36:46.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:46.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:46.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.776402+0000", , 2022-09-04T14:36:46.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:46.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:46.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.777099+0000", , 2022-09-04T14:36:46.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:46.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:46.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.777129+0000", , 2022-09-04T14:36:46.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:46.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:46.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.796813+0000", , 2022-09-04T14:36:46.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1941e-05 , 2022-09-04T14:36:46.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:47.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.067536+0000", , 2022-09-04T14:36:47.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4384e-05 , 2022-09-04T14:36:47.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:47.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.067574+0000", , 2022-09-04T14:36:47.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5220000000000001e-06 , 2022-09-04T14:36:47.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:47.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069459+0000", , 2022-09-04T14:36:47.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0716e-05 , 2022-09-04T14:36:47.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:47.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.080247+0000", , 2022-09-04T14:36:47.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0058263680000000002 , 2022-09-04T14:36:47.091 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:47.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:59.058232+0000", , 2022-09-04T14:36:47.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000153172 , 2022-09-04T14:36:47.113 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:47.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.176512+0000", , 2022-09-04T14:36:47.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022595950000000001 , 2022-09-04T14:36:47.135 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:47.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.176837+0000", , 2022-09-04T14:36:47.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002025481 , 2022-09-04T14:36:47.158 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:48.776+0000 7fdfc7e2d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 39182) UID: 0 , 2022-09-04T14:36:48.878+0000 7faf5177f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38712) UID: 0 , 2022-09-04T14:36:48.886 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: osd.1: osd_enable_op_tracker = 'true' , osd.2: osd_enable_op_tracker = 'true' , osd.3: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:36:50.716 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:50.716 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.7s3 47/33 ECSubWrite(tid=80, reqid=client.4557.0:287, at_version=0'0, trim_to=0'0, roll_forward_to=47'25))", , 2022-09-04T14:36:50.716 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:23.818244+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:50.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.842593+0000", , 2022-09-04T14:36:50.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6279000000000001e-05 , 2022-09-04T14:36:50.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:50.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:50.773 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:50.773 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.9s2 53/33 ECSubWrite(tid=65, reqid=client.4557.0:352, at_version=53'47, trim_to=0'0, roll_forward_to=48'44))", Ignoring malparsed backtrace: 2022-09-04T14:36:50.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:36:50.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682584+0000", , 2022-09-04T14:36:50.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7893e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:50.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:50.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910693+0000", , 2022-09-04T14:36:50.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000128956 Ignoring malparsed backtrace: 2022-09-04T14:36:50.890 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:50.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:50.891 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:50.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", , 2022-09-04T14:36:50.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:36:50.917 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 589 Ignoring malparsed backtrace: 2022-09-04T14:36:50.946 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:50.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:36:50.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:46.868685+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:50.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:50.977 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:50.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:36:51.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.002 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:51.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:36:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:36:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9819999999999996e-06 , 2022-09-04T14:36:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7813999999999997e-05 , 2022-09-04T14:36:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1012e-05 , 2022-09-04T14:36:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:51.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.723988+0000", , 2022-09-04T14:36:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001210769 , 2022-09-04T14:36:51.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.737747+0000", , 2022-09-04T14:36:51.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081521979999999994 , 2022-09-04T14:36:51.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:51.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.738160+0000", , 2022-09-04T14:36:51.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019483400000000001 , 2022-09-04T14:36:51.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:51.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018782047999999999, , 2022-09-04T14:36:51.692 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:51.692 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:36:51.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019980041, , 2022-09-04T14:36:51.716 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:51.716 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", Ignoring malparsed backtrace: 2022-09-04T14:36:51.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:36:51.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.773225+0000", , 2022-09-04T14:36:51.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6647000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", , 2022-09-04T14:36:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834285+0000", , 2022-09-04T14:36:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010179999999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:51.785 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.786 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:36:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.909081+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.911183+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:36:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:51.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.882 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:51.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:51.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.906 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:51.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for blocked object", Ignoring malparsed backtrace: 2022-09-04T14:36:51.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.932 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:51.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:51.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:51.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:51.965 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:51.999 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:36:51.999 INFO:teuthology.orchestra.run.smithi038.stdout:} , 2022-09-04T14:36:52.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight Ignoring malparsed backtrace: 2022-09-04T14:36:52.523 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:52.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:52.524 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:52.550 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:52.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:36:52.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682487+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:52.578 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:52.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:36:52.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.683129+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:52.603 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:36:52.604 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:36:52.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:52.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:52.629 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:52.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:36:52.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:52.654 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:36:52.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:36:52.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:36:52.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:52.680 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:52.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4665000000000001e-05 , 2022-09-04T14:36:52.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:52.706 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:52.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4836e-05 , 2022-09-04T14:36:52.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:36:52.731 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:52.756 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:52.756 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:52.756 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:52.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.129740+0000", , 2022-09-04T14:36:52.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6159999999999999e-05 , 2022-09-04T14:36:52.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:52.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.132839+0000", , 2022-09-04T14:36:52.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0402e-05 , 2022-09-04T14:36:52.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:52.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012410411, , 2022-09-04T14:36:52.838 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:52.839 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:36:52.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010350293, , 2022-09-04T14:36:52.864 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:36:52.864 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:36:52.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:36:52.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:16.172707+0000", , 2022-09-04T14:36:52.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4805999999999998e-05 Ignoring malparsed backtrace: osd.5: osd_enable_op_tracker = 'false' , osd.6: osd_enable_op_tracker = 'false' , osd.7: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:36:55.918 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} , 2022-09-04T14:36:55.932+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:36:55.940 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} Ignoring malparsed backtrace: 2022-09-04T14:36:56.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:56.160 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:56.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:36:56.196 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:80 3.5s0 3:a9d45bed:::smithi16339483-27:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:36:56.197 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.765917+0000", , 2022-09-04T14:36:56.197 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 163.35306787299999, Ignoring malparsed backtrace: 2022-09-04T14:36:56.219 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:81 3.5s0 3:a9d45bed:::smithi16339483-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:36:56.220 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.765934+0000", , 2022-09-04T14:36:56.220 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 163.353050639, Ignoring malparsed backtrace: 2022-09-04T14:36:56.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:56.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:56.242 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:56.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9360000000000001e-06 , 2022-09-04T14:36:56.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:56.264 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:56.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2220000000000001e-06 , 2022-09-04T14:36:56.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:56.287 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:56.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.669e-05 , 2022-09-04T14:36:56.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:56.309 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:56.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4320000000000006e-05 , 2022-09-04T14:36:56.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:56.331 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:56.353 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:36:56.353 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:36:56.354 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:36:56.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784610+0000", , 2022-09-04T14:36:56.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0583999999999997e-05 , 2022-09-04T14:36:56.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:56.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.785111+0000", , 2022-09-04T14:36:56.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.861e-06 , 2022-09-04T14:36:56.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:56.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0083998100000000006, , 2022-09-04T14:36:56.420 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:36:56.420 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:36:56.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.011064098, , 2022-09-04T14:36:56.442 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:36:56.443 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:36:56.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53432486700000004, , 2022-09-04T14:36:56.465 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:36:56.465 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:36:56.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:36:56.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.174801+0000", , 2022-09-04T14:36:56.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001100932 Ignoring malparsed backtrace: 2022-09-04T14:36:56.509 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:56.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:21.266460+0000", , 2022-09-04T14:36:56.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2180000000000005e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:56.533 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} , 2022-09-04T14:36:56.531+0000 7fbf31779700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 38722) UID: 0 , 2022-09-04T14:36:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 Ignoring malparsed backtrace: 2022-09-04T14:36:56.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:56.995 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:56.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:36:57.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:57.023 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:57.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:36:57.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:57.048 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:57.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:57.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:36:57.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:57.075 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:57.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3650000000000004e-06 , 2022-09-04T14:36:57.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:57.100 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:57.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7658999999999998e-05 , 2022-09-04T14:36:57.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:57.125 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:57.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011705900000000001 , 2022-09-04T14:36:57.161 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:36:57.161 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:36:57.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.914081+0000", , 2022-09-04T14:36:57.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:36:57.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:57.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:36:57.212 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:57.212 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:571 3.cs0 3:36aacf6f:::smithi16339483-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", Ignoring malparsed backtrace: 2022-09-04T14:36:57.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:36:57.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949095+0000", , 2022-09-04T14:36:57.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.044e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:57.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:36:57.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.953345+0000", , 2022-09-04T14:36:57.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011661499999999999 Ignoring malparsed backtrace: 2022-09-04T14:36:57.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:36:57.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.954158+0000", , 2022-09-04T14:36:57.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103896 Ignoring malparsed backtrace: 2022-09-04T14:36:57.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:36:57.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069715+0000", , 2022-09-04T14:36:57.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6446000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:36:57.343 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:57.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:36:57.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071671+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:57.375 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:57.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:57.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119221+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:57.401 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:36:57.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:36:57.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.176076+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:57.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3949000000000003e-05 , 2022-09-04T14:36:57.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:57.919 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:57.942 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0066411739999999997 , 2022-09-04T14:36:57.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:57.943 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:57.968 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:57.968 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:309 3.2s1 3:4f5d92c8:::smithi16339483-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", , 2022-09-04T14:36:57.968 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.841116+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:57.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843185+0000", , 2022-09-04T14:36:57.992 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010238699999999999 , 2022-09-04T14:36:57.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:58.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680947+0000", , 2022-09-04T14:36:58.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00071366400000000003 , 2022-09-04T14:36:58.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:58.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683148+0000", , 2022-09-04T14:36:58.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00078775199999999996 , 2022-09-04T14:36:58.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:58.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683684+0000", , 2022-09-04T14:36:58.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010369000000000001 , 2022-09-04T14:36:58.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:36:58.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.087235+0000", , 2022-09-04T14:36:58.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028045349999999999 , 2022-09-04T14:36:58.084 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:36:58.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:36:58.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.069368+0000", , 2022-09-04T14:36:58.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:36:58.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:36:58.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046768+0000", , 2022-09-04T14:36:58.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4300000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:58.153 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:36:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:58.154 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:36:58.177 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.065918+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.200 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:36:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.223 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:36:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:58.248 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:36:58.272 INFO:teuthology.orchestra.run.smithi163.stdout:} , 2022-09-04T14:36:58.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight , 2022-09-04T14:36:58.272+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:36:58.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:36:58.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.734051+0000", , 2022-09-04T14:36:58.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8319999999999995e-06 Ignoring malparsed backtrace: 2022-09-04T14:36:58.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:36:58.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.751044+0000", , 2022-09-04T14:36:58.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0087759959999999995 Ignoring malparsed backtrace: 2022-09-04T14:36:58.801 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.801 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:36:58.801 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.774974+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.825 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.825 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:36:58.825 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.775016+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.848 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.849 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:36:58.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778520+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.872 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.872 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:36:58.872 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.780952+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.896 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.896 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:36:58.896 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.781059+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.919 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.920 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:36:58.920 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.791872+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.943 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.943 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:36:58.943 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.808834+0000", Ignoring malparsed backtrace: 2022-09-04T14:36:58.966 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:18.787757+0000", , 2022-09-04T14:36:58.967 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 159.93474066100001, , 2022-09-04T14:36:58.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.079647366999999997, Ignoring malparsed backtrace: 2022-09-04T14:36:58.990 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:36:58.990 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:58.991 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:36:59.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:59.014 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:36:59.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:36:59.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:36:59.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:59.038 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:59.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:36:59.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:59.061 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:59.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4295e-05 , 2022-09-04T14:36:59.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:59.085 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:59.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 , 2022-09-04T14:36:59.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:59.109 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:36:59.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2537e-05 , 2022-09-04T14:36:59.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:36:59.133 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:00.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops , nodeep-scrub is set , 2022-09-04T14:37:00.274+0000 7fc80fcb9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 39177) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:37:01.537 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:01.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:37:01.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:13.003952+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:01.563 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:01.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:01.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.774759+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:01.590 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:01.591 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:01.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:01.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:01.618 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:01.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:01.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:01.643 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:01.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:01.665 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:01.665 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:01.666 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:01.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:37:01.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:01.688 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:01.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5889999999999993e-05 , 2022-09-04T14:37:01.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:01.711 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:01.735 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:37:01.735 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:01.735 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:37:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.090411+0000", , 2022-09-04T14:37:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0028999999999999e-05 , 2022-09-04T14:37:01.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:01.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126684+0000", , 2022-09-04T14:37:01.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1969e-05 , 2022-09-04T14:37:01.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:01.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014925337, , 2022-09-04T14:37:01.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:37:01.807 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:01.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:37:01.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.133736+0000", , 2022-09-04T14:37:01.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8424000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:37:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:10.128685+0000", , 2022-09-04T14:37:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001356884 Ignoring malparsed backtrace: 2022-09-04T14:37:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:16.172089+0000", , 2022-09-04T14:37:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5874999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:01.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} , 2022-09-04T14:37:01.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} , 2022-09-04T14:37:01.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} Ignoring malparsed backtrace: 2022-09-04T14:37:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783486+0000", , 2022-09-04T14:37:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7790000000000007e-06 , 2022-09-04T14:37:02.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084550000000000007, , 2022-09-04T14:37:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:37:02.403 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:37:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833159+0000", , 2022-09-04T14:37:02.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9927999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:02.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834098+0000", , 2022-09-04T14:37:02.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1070000000000005e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 154.58257639999999, , 2022-09-04T14:37:02.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094122489999999993, , 2022-09-04T14:37:02.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:37:02.502 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:37:02.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.778871+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:02.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.023108+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:39.909986+0000", , 2022-09-04T14:37:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 142.44979427199999, , 2022-09-04T14:37:02.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027652264999999999, Ignoring malparsed backtrace: 2022-09-04T14:37:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:02.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:02.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:37:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.cs5 69/33 ECSubWrite(tid=45, reqid=client.4557.0:581, at_version=69'33, trim_to=0'0, roll_forward_to=61'29))", , 2022-09-04T14:37:02.621 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:45.954801+0000", , 2022-09-04T14:37:02.622 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 136.404979158, Ignoring malparsed backtrace: 2022-09-04T14:37:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012455300000000001 , 2022-09-04T14:37:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:02.645 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:02.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001393 , 2022-09-04T14:37:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:02.692 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.bs3 92/33 ECSubWrite(tid=263, reqid=client.4557.0:859, at_version=0'0, trim_to=0'0, roll_forward_to=92'50))", , 2022-09-04T14:37:02.693 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:35:09.129949+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:17.175748+0000", , 2022-09-04T14:37:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8968000000000001e-05 , 2022-09-04T14:37:02.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:03.194 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:03.194 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:03.204 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:03.229 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:03.229 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:03.230 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:37:03.260 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.bs2 42/33 ECSubWrite(tid=59, reqid=client.4557.0:197, at_version=42'11, trim_to=0'0, roll_forward_to=36'7))", , 2022-09-04T14:37:03.260 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:18.784575+0000", , 2022-09-04T14:37:03.260 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 164.395711877, Ignoring malparsed backtrace: 2022-09-04T14:37:03.285 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.5189e-05 , 2022-09-04T14:37:03.285 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:03.285 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:03.309 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00012498199999999999 , 2022-09-04T14:37:03.309 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:03.309 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:03.333 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:03.333 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.9s1 42/33 ECSubWrite(tid=31, reqid=client.4557.0:207, at_version=42'19, trim_to=0'0, roll_forward_to=37'16))", , 2022-09-04T14:37:03.333 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:18.790361+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:03.357 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.805469+0000", , 2022-09-04T14:37:03.357 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.6536000000000001e-05 , 2022-09-04T14:37:03.357 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:03.381 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816859+0000", , 2022-09-04T14:37:03.381 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00019410599999999999 , 2022-09-04T14:37:03.382 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:03.406 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:03.406 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:03.406 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.9s1 53/33 ECSubWrite(tid=66, reqid=client.4557.0:353, at_version=53'48, trim_to=0'0, roll_forward_to=48'44))", Ignoring malparsed backtrace: 2022-09-04T14:37:03.429 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:37:03.430 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:33.682936+0000", , 2022-09-04T14:37:03.430 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.9345999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:03.453 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:03.454 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:51.083908+0000", , 2022-09-04T14:37:03.454 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00012517100000000001 Ignoring malparsed backtrace: 2022-09-04T14:37:03.478 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:37:03.478 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:03.478 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:03.501 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:03.502 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:37:03.502 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:09.131960+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:03.525 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:03.525 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:03.526 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:21.265589+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:03.549 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:37:03.549 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:37:03.550 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:03.885+0000 7fbf31779700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 38722) UID: 0 , 2022-09-04T14:37:03.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false , 2022-09-04T14:37:03.969 INFO:teuthology.orchestra.run.smithi077.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:37:03.994 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:37:03.995 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.809261+0000", , 2022-09-04T14:37:03.995 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:37:04.024 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:37:04.025 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831709+0000", , 2022-09-04T14:37:04.025 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2118e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:04.048 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:37:04.049 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.841429+0000", , 2022-09-04T14:37:04.049 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0047505940000000003 Ignoring malparsed backtrace: 2022-09-04T14:37:04.073 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:04.073 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:37:04.074 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.835466+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:04.105 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:04.106 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:37:04.106 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.842400+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:04.134 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:04.134 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:37:04.134 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.849316+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:04.160 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:04.161 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:04.161 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:04.185 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:04.185 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:04.185 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:04.208 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:04.208 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:04.208 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:04.232 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:04.232 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:04.232 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:04.255 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.7998000000000003e-05 , 2022-09-04T14:37:04.255 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:04.256 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:04.279 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.0358999999999994e-05 , 2022-09-04T14:37:04.279 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:04.279 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:04.302 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.074301+0000", , 2022-09-04T14:37:04.303 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:04.303 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:04.326 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.103537+0000", , 2022-09-04T14:37:04.326 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.9397999999999997e-05 , 2022-09-04T14:37:04.326 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:04.349 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.105722+0000", , 2022-09-04T14:37:04.349 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2036e-05 , 2022-09-04T14:37:04.349 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:05.966 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-13, ['dump_ops_in_flight'] , 2022-09-04T14:37:05.966+0000 7fbf31779700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 38722) UID: 0 , 2022-09-04T14:37:05.990+0000 7f403142b700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 Ignoring malparsed backtrace: 2022-09-04T14:37:07.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.088 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:07.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:07.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.113 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:07.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:07.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.137 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:07.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:07.160 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:07.160 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:37:07.161 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:07.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2919999999999998e-06 , 2022-09-04T14:37:07.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.189 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:07.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8559999999999998e-06 , 2022-09-04T14:37:07.213 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.213 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:07.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112991 , 2022-09-04T14:37:07.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.246 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:07.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019956999999999999 , 2022-09-04T14:37:07.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.270 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:07.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110553 , 2022-09-04T14:37:07.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.298 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:07.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046751+0000", , 2022-09-04T14:37:07.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:37:07.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:07.346 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046906+0000", , 2022-09-04T14:37:07.346 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107398 , 2022-09-04T14:37:07.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:07.368 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:37:07.369 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.052267+0000", , 2022-09-04T14:37:07.369 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7239999999999998e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:07.392 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:07.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:07.393 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:07.416 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:07.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:37:07.416 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173190+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:07.440 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:07.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", , 2022-09-04T14:37:07.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.247583+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:07.462 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:07.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", , 2022-09-04T14:37:07.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.264100+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:08.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.728030+0000", , 2022-09-04T14:37:08.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9370000000000004e-06 , 2022-09-04T14:37:08.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:08.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.728054+0000", , 2022-09-04T14:37:08.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1450000000000002e-06 , 2022-09-04T14:37:08.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:08.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.763558+0000", , 2022-09-04T14:37:08.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078671650000000006 , 2022-09-04T14:37:08.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:08.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778528+0000", , 2022-09-04T14:37:08.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014076799999999999 , 2022-09-04T14:37:08.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:08.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.791253+0000", , 2022-09-04T14:37:08.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019795580000000002 , 2022-09-04T14:37:08.099 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:08.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013839605, , 2022-09-04T14:37:08.123 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:37:08.123 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:37:08.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016511014000000001, , 2022-09-04T14:37:08.147 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:37:08.147 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", Ignoring malparsed backtrace: 2022-09-04T14:37:08.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:37:08.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.782314+0000", , 2022-09-04T14:37:08.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:37:08.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:37:08.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.782342+0000", , 2022-09-04T14:37:08.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:37:08.218 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:37:08.218 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.822936+0000", , 2022-09-04T14:37:08.218 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001483159 Ignoring malparsed backtrace: 2022-09-04T14:37:08.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:08.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.791746+0000", , 2022-09-04T14:37:08.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8689999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:08.265 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 169.19804726500001, , 2022-09-04T14:37:08.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.075388408000000004, , 2022-09-04T14:37:08.266 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:37:08.288 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:08.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:08.289 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.912884+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:08.312 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:08.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", , 2022-09-04T14:37:08.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.690847+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:08.335 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:08.336 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:08.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:08.359 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:08.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:37:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:08.384 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:37:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:37:08.885 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:37:08.911 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:37:08.911 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:37:08.912 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:37:08.937 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:08.937 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:08.938 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:08.965 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:08.965 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:08.965 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:37:08.989 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.5s6 47/33 ECSubWrite(tid=21, reqid=client.4557.0:300, at_version=47'15, trim_to=0'0, roll_forward_to=37'11))", , 2022-09-04T14:37:08.989 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.833747+0000", , 2022-09-04T14:37:08.990 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 165.045272195, Ignoring malparsed backtrace: 2022-09-04T14:37:09.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2073e-05 , 2022-09-04T14:37:09.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:09.017 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:09.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9834999999999996e-05 , 2022-09-04T14:37:09.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:09.047 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:09.071 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:09.072 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs5 81/52 ECSubWrite(tid=81, reqid=client.4557.0:690, at_version=81'49, trim_to=0'0, roll_forward_to=71'47))", , 2022-09-04T14:37:09.072 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:58.086174+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:09.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.090362+0000", , 2022-09-04T14:37:09.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4104000000000001e-05 , 2022-09-04T14:37:09.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:09.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.056115+0000", , 2022-09-04T14:37:09.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020950400000000001 , 2022-09-04T14:37:09.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:09.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.063701+0000", , 2022-09-04T14:37:09.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7571000000000002e-05 , 2022-09-04T14:37:09.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:09.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:09.166 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.170924+0000", , 2022-09-04T14:37:09.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:09.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:09.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.172955+0000", , 2022-09-04T14:37:09.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7799999999999995e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:09.214 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:09.214 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173525+0000", , 2022-09-04T14:37:09.214 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9571000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:09.237 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:09.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:09.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.175739+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:09.270 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:09.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:09.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:22.192292+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:09.740 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:09.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:37:09.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.795362+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:09.778 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:09.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:09.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.803055+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:09.804 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:09.805 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:09.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:09.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:09.831 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:09.831 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:09.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:09.862 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:09.863 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:09.892 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:09.892 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:09.892 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:09.919 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:37:09.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:09.920 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:09.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2001000000000001e-05 , 2022-09-04T14:37:09.944 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:09.944 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:09.967 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0074529050000000001 , 2022-09-04T14:37:09.967 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:09.967 INFO:teuthology.orchestra.run.smithi163.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:37:09.990 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:45.963781+0000", , 2022-09-04T14:37:09.990 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 , 2022-09-04T14:37:09.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:10.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:45.967328+0000", , 2022-09-04T14:37:10.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9390000000000003e-06 , 2022-09-04T14:37:10.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:10.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.092433+0000", , 2022-09-04T14:37:10.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026217670000000001 , 2022-09-04T14:37:10.042 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:10.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:37:10.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.172633+0000", , 2022-09-04T14:37:10.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 Ignoring malparsed backtrace: 2022-09-04T14:37:10.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:37:10.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173140+0000", , 2022-09-04T14:37:10.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.96e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:10.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:37:10.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.191397+0000", , 2022-09-04T14:37:10.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015595278000000001 Ignoring malparsed backtrace: 2022-09-04T14:37:09.946+0000 7f0a41d9b700 0 -- 172.21.15.38:0/265780507 >> [v2:172.21.15.163:6808/38951,v1:172.21.15.163:6809/38951] conn(0x7f0a44218ad0 msgr2=0x7f0a44218f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure , 2022-09-04T14:37:10.141 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} , 2022-09-04T14:37:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 Ignoring malparsed backtrace: 2022-09-04T14:37:12.255+0000 7fdfc7e2d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 39182) UID: 0 , 2022-09-04T14:37:12.261 INFO:teuthology.orchestra.run.smithi077.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:37:12.261 INFO:teuthology.orchestra.run.smithi077.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2022-09-04T14:37:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:12.467 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:12.468 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:12.493 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.6829999999999996e-06 , 2022-09-04T14:37:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:12.494 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.2556999999999999e-05 , 2022-09-04T14:37:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:12.519 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.787916+0000", , 2022-09-04T14:37:12.545 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:12.546 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.789611+0000", , 2022-09-04T14:37:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.2710000000000003e-06 , 2022-09-04T14:37:12.571 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.792867+0000", , 2022-09-04T14:37:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0020676370000000002 , 2022-09-04T14:37:12.597 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:37:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816324+0000", , 2022-09-04T14:37:12.626 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:12.650 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686083+0000", , 2022-09-04T14:37:12.651 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.4999999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686495+0000", , 2022-09-04T14:37:12.679 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.9624e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:12.710 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:37:12.711 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:51.083397+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:12.737 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:12.738 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.069466+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:12.765 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.073052+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:37:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:12.788 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:12.810 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:12.811 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:12.832 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:12.833 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:13.249 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:37:13.250 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:13.250 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:13.290 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.268e-06 , 2022-09-04T14:37:13.291 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:13.291 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:13.324 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00020845999999999999 , 2022-09-04T14:37:13.324 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:37:13.325 INFO:teuthology.orchestra.run.smithi077.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:37:13.354 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834388+0000", , 2022-09-04T14:37:13.354 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.9999986 , 2022-09-04T14:37:13.354 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:13.382 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:24.598449+0000", , 2022-09-04T14:37:13.383 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.0086e-05 , 2022-09-04T14:37:13.383 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:13.413 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.848127+0000", , 2022-09-04T14:37:13.413 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0040956559999999996 , 2022-09-04T14:37:13.413 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:13.440 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:37:13.441 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843584+0000", , 2022-09-04T14:37:13.441 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:37:13.468 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:37:13.468 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684739+0000", , 2022-09-04T14:37:13.469 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 9.3940000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:37:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.689591+0000", , 2022-09-04T14:37:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0046552959999999997 Ignoring malparsed backtrace: 2022-09-04T14:37:13.523 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:13.523 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:37:13.523 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.968392+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:13.550 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:13.550 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:37:13.550 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971484+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:13.580 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:13.580 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:37:13.581 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.979118+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:13.609 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:13.609 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:13.610 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:13.637 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:13.637 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:13.637 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:13.664 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:13.664 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:13.664 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:13.993 INFO:teuthology.orchestra.run.smithi077.stdout: "num_blocked_ops": 0 , 2022-09-04T14:37:13.994 INFO:teuthology.orchestra.run.smithi077.stdout:} , 2022-09-04T14:37:13.991+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:37:14.153 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:14.153 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:14.153 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:14.176 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.9429999999999999e-05 , 2022-09-04T14:37:14.176 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:14.176 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:14.198 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.5996999999999999e-05 , 2022-09-04T14:37:14.199 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:14.199 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:14.221 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831072+0000", , 2022-09-04T14:37:14.221 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:14.221 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:14.244 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833071+0000", , 2022-09-04T14:37:14.244 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 9.4210999999999996e-05 , 2022-09-04T14:37:14.244 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:14.266 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833746+0000", , 2022-09-04T14:37:14.267 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00038284199999999999 , 2022-09-04T14:37:14.267 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:14.289 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:37:14.289 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834171+0000", , 2022-09-04T14:37:14.289 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:14.312 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:37:14.312 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971520+0000", , 2022-09-04T14:37:14.312 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.8488000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:14.334 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:14.335 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973486+0000", , 2022-09-04T14:37:14.335 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.1992e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:14.357 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:14.357 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:37:14.358 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.063496+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:14.380 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:14.380 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:37:14.381 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.075997+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:14.403 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:14.403 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:14.403 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.082223+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:14.427 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:14.427 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:14.428 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:37:14.454 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:14.454 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:14.454 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:14.483 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:14.483 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:14.484 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:37:14.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:37:14.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828247+0000", , 2022-09-04T14:37:14.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00089402299999999997 Ignoring malparsed backtrace: 2022-09-04T14:37:14.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:14.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.829645+0000", , 2022-09-04T14:37:14.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1580000000000004e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:15.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:15.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.830397+0000", , 2022-09-04T14:37:15.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6013000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:15.036 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:37:15.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.813732+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:15.060 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:37:15.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.829657+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:37:15.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.838402+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:15.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.107 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:15.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.131 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:15.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.155 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:15.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.179 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:15.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.203 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:15.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.227 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:15.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.251 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:15.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.275 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:15.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:37:15.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.299 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:37:15.299 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:15.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.322 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:37:15.322 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:15.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:15.352 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:37:15.353 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:15.380 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} , 2022-09-04T14:37:15.380 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} , 2022-09-04T14:37:15.380 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} Ignoring malparsed backtrace: 2022-09-04T14:37:17.179+0000 7f4035433700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 , 2022-09-04T14:37:17.193+0000 7fdfc7e2d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 39182) UID: 0 , 2022-09-04T14:37:17.208 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:37:17.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:17.982 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:17.982 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:18.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.019 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:18.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.055 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:37:18.098 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.5s6 47/33 ECSubWrite(tid=20, reqid=client.4557.0:299, at_version=47'14, trim_to=0'0, roll_forward_to=37'11))", , 2022-09-04T14:37:18.098 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.833001+0000", , 2022-09-04T14:37:18.098 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 174.13573960400001, Ignoring malparsed backtrace: 2022-09-04T14:37:18.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5441e-05 , 2022-09-04T14:37:18.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.122 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:18.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9816999999999997e-05 , 2022-09-04T14:37:18.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.146 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:18.169 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.169 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.5s6 47/33 ECSubWrite(tid=22, reqid=client.4557.0:300, at_version=0'0, trim_to=0'0, roll_forward_to=47'15))", , 2022-09-04T14:37:18.169 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.838592+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.088668+0000", , 2022-09-04T14:37:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002494886 , 2022-09-04T14:37:18.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:18.216 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.090540+0000", , 2022-09-04T14:37:18.216 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0043999999999995e-05 , 2022-09-04T14:37:18.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:18.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.239 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.239 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:712 3.es0 3:70fb2152:::smithi16339483-29:head [append 0~507904] snapc 2b=[2b,27,23,14] ondisk+write+known_if_redirected+supports_pool_eio e82)", Ignoring malparsed backtrace: 2022-09-04T14:37:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.226201+0000", , 2022-09-04T14:37:18.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:18.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.171035+0000", , 2022-09-04T14:37:18.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2014000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173450+0000", , 2022-09-04T14:37:18.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000292565 Ignoring malparsed backtrace: 2022-09-04T14:37:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:18.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175435+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.357 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:18.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.175885+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:18.381 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:18.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:22.192595+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.800785+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:18.905 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:37:18.906 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.906 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:18.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.928 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.929 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:18.951 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.951 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:18.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:18.973 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:18.973 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:37:18.974 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:18.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015109699999999999 , 2022-09-04T14:37:18.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:18.997 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:19.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00054069899999999995 , 2022-09-04T14:37:19.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:19.028 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:19.064 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:37:19.064 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:19.065 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:37:19.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:41.959946+0000", , 2022-09-04T14:37:19.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3825e-05 , 2022-09-04T14:37:19.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:19.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:45.964130+0000", , 2022-09-04T14:37:19.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.046e-05 , 2022-09-04T14:37:19.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:19.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057242150000000004, , 2022-09-04T14:37:19.140 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:37:19.141 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:19.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:37:19.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.170627+0000", , 2022-09-04T14:37:19.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7114000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:19.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:19.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173100+0000", , 2022-09-04T14:37:19.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1732999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:19.211 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 122.68978699500001, , 2022-09-04T14:37:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015936466999999999, , 2022-09-04T14:37:19.212 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:37:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:19.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:37:19.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.271794+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:19.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:37:19.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.828552+0000", , 2022-09-04T14:37:19.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7828e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:19.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:19.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.831711+0000", , 2022-09-04T14:37:19.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121205 Ignoring malparsed backtrace: 2022-09-04T14:37:19.752 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:19.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:19.753 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:19.776 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:19.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:19.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682919+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:19.800 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:19.801 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:19.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:19.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:19.824 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:19.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:19.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:37:19.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:19.872 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:19.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:37:19.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3239999999999997e-06 , 2022-09-04T14:37:19.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:19.896 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:19.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0988000000000001e-05 , 2022-09-04T14:37:19.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:19.920 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:19.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077956+0000", , 2022-09-04T14:37:19.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:19.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:19.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126225+0000", , 2022-09-04T14:37:19.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4749999999999999e-06 , 2022-09-04T14:37:19.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:19.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126582+0000", , 2022-09-04T14:37:19.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6290000000000001e-05 , 2022-09-04T14:37:19.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:20.014 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:37:20.015 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 932 , 2022-09-04T14:37:20.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:20.038 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:37:20.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 933 , 2022-09-04T14:37:20.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:20.262+0000 7faa2b696700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{2b=[2b,27,23,14],33=[33,2f],43=[43,42]} for object 3:70fb2152:::smithi16339483-29:9 , 2022-09-04T14:37:20.262+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:37:20.262+0000 7f4e01c5f700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{c=[c],15=[15,14,12,11,e],1e=[1e,1b,17],2b=[27,23],43=[42]} for object 3:38890410:::smithi16339483-19:9 Ignoring malparsed backtrace: 2022-09-04T14:37:21.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight , 2022-09-04T14:37:21.596+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 , 2022-09-04T14:37:21.676+0000 7f4e208b2700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 38721) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:37:23.351 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:37:23.351 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:37:23.351 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:37:23.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.378 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:23.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:37:23.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.418 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:23.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:23.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.449 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:23.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:23.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.472 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:23.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:23.496 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:37:23.496 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:37:23.496 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:37:23.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022150249999999998 , 2022-09-04T14:37:23.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.520 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:23.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.839e-06 , 2022-09-04T14:37:23.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.545 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:23.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.181e-06 , 2022-09-04T14:37:23.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.568 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:23.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.181e-06 , 2022-09-04T14:37:23.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:23.591 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:23.616 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:37:23.616 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:23.616 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:37:23.642 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:37:23.642 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:37:23.643 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:37:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:37:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:37:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:37:23.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069239+0000", , 2022-09-04T14:37:23.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:37:23.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:23.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071748+0000", , 2022-09-04T14:37:23.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4035e-05 , 2022-09-04T14:37:23.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:23.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:59.052156+0000", , 2022-09-04T14:37:23.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010442159999999999 , 2022-09-04T14:37:23.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:23.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.171638+0000", , 2022-09-04T14:37:23.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00074510199999999998 , 2022-09-04T14:37:23.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:23.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:16.172590+0000", , 2022-09-04T14:37:23.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058582999999999997 , 2022-09-04T14:37:23.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:24.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:37:24.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.721955+0000", , 2022-09-04T14:37:24.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:24.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:24.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.722273+0000", , 2022-09-04T14:37:24.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0345e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:24.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:37:24.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.733858+0000", , 2022-09-04T14:37:24.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6212999999999996e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:24.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:24.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.762422+0000", , 2022-09-04T14:37:24.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4604000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:24.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:37:24.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.785233+0000", , 2022-09-04T14:37:24.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024086218999999999 Ignoring malparsed backtrace: 2022-09-04T14:37:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:37:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.759683+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:24.438 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:37:24.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.785979+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:24.462 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:37:24.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.845676+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:24.485 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:37:24.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.847370+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:24.509 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:37:24.510 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:24.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:24.534 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:24.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:24.558 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:37:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.as0 81/33 ECSubWriteReply(tid=123, last_complete=81'58, committed=1, applied=1))", , 2022-09-04T14:37:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:58.091529+0000", , 2022-09-04T14:37:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 146.163904597, Ignoring malparsed backtrace: 2022-09-04T14:37:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 97/33 ECSubWriteReply(tid=161, last_complete=97'57, committed=1, applied=1))", , 2022-09-04T14:37:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:35:14.246843+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:24.676 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:24.676 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:37:24.676 INFO:teuthology.orchestra.run.smithi155.stdout:} Ignoring malparsed backtrace: 2022-09-04T14:37:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.729458+0000", , 2022-09-04T14:37:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016161100000000001 , 2022-09-04T14:37:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.742375+0000", , 2022-09-04T14:37:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011993850000000001 , 2022-09-04T14:37:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:25.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.976768+0000", , 2022-09-04T14:37:25.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8718000000000003e-05 , 2022-09-04T14:37:25.193 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:25.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:37:25.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.788724+0000", , 2022-09-04T14:37:25.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:25.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:37:25.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.789269+0000", , 2022-09-04T14:37:25.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:37:25.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:37:25.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.789290+0000", , 2022-09-04T14:37:25.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:37:25.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:37:25.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858490+0000", , 2022-09-04T14:37:25.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3427000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:25.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:37:25.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.866064+0000", , 2022-09-04T14:37:25.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0073323850000000003 Ignoring malparsed backtrace: 2022-09-04T14:37:25.331 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:25.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:25.331 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:25.355 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:25.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:37:25.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784579+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:25.378 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:25.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:25.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.785104+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:25.410 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:37:25.411 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:25.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:25.437 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:37:25.437 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:25.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:25.460 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:37:25.460 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:25.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:25.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:25.484 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:25.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", Ignoring malparsed backtrace: 2022-09-04T14:37:25.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:25.507 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:25.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:25.531 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:25.531 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:37:25.531 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:26.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7949999999999997e-05 , 2022-09-04T14:37:26.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:26.001 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:26.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1889000000000002e-05 , 2022-09-04T14:37:26.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:26.030 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:26.060 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.060 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:158 3.cs0 3:38890410:::smithi16339483-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:37:26.061 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.858431+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801791+0000", , 2022-09-04T14:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.815527+0000", , 2022-09-04T14:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2998e-05 , 2022-09-04T14:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.824067+0000", , 2022-09-04T14:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0071055270000000004 , 2022-09-04T14:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.06678133, , 2022-09-04T14:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:26.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:37:26.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:24.340468+0000", , 2022-09-04T14:37:26.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.42638076499999999 Ignoring malparsed backtrace: 2022-09-04T14:37:26.214 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:37:26.214 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:37:26.215 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 571 Ignoring malparsed backtrace: 2022-09-04T14:37:26.239 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:26.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949105+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:26.263 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:26.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.954003+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:26.288 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:26.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.954711+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:26.312 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:37:26.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.079765+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:26.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:26.337 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:26.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:26.361 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:26.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:26.385 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:26.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:27.910 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:37:27.910 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:37:27.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:37:28.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020747589999999998 , 2022-09-04T14:37:28.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:28.762 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:28.787 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:37:28.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:28.788 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:37:28.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:24.311442+0000", , 2022-09-04T14:37:28.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.44929617900000002 , 2022-09-04T14:37:28.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:28.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.05929102, , 2022-09-04T14:37:28.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:37:28.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:28.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:37:28.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910470+0000", , 2022-09-04T14:37:28.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5518000000000004e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:28.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:28.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910992+0000", , 2022-09-04T14:37:28.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4729999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", , 2022-09-04T14:37:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:46.868754+0000", , 2022-09-04T14:37:28.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.90345453600000003 Ignoring malparsed backtrace: 2022-09-04T14:37:28.930 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:28.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:37:28.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.965465+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:28.953 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:28.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:37:28.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.061465+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:28.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:29.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.001 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:29.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:29.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:29.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:29.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.050 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:29.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0672999999999999e-05 , 2022-09-04T14:37:29.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.075 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:29.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0730000000000001e-06 , 2022-09-04T14:37:29.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.099 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:29.584 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ , 2022-09-04T14:37:29.584 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:29.585 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:12 3.7s0 3:ebb7004a:::smithi16339483-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", Ignoring malparsed backtrace: 2022-09-04T14:37:29.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:37:29.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.724116+0000", , 2022-09-04T14:37:29.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:29.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:29.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.728527+0000", , 2022-09-04T14:37:29.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8250000000000004e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:29.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:29.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.728548+0000", , 2022-09-04T14:37:29.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6450000000000004e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:29.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", , 2022-09-04T14:37:29.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.752762+0000", , 2022-09-04T14:37:29.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2735e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:29.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", , 2022-09-04T14:37:29.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.753452+0000", , 2022-09-04T14:37:29.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9328999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:29.737 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 185.74974045100001, , 2022-09-04T14:37:29.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.76853404199999997, , 2022-09-04T14:37:29.738 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:37:29.760 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:29.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:37:29.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.860979+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:29.794 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:29.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:29.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.780274+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:29.817 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:39.910370+0000", , 2022-09-04T14:37:29.817 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 169.66992687999999, , 2022-09-04T14:37:29.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021925577000000002, Ignoring malparsed backtrace: 2022-09-04T14:37:29.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.839 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:29.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:29.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.862 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:29.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:29.884 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:709 3.1s0 3:89c1087a:::smithi16339483-13:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", , 2022-09-04T14:37:29.885 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:59.046836+0000", , 2022-09-04T14:37:29.885 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 150.533461085, Ignoring malparsed backtrace: 2022-09-04T14:37:29.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9470000000000003e-06 , 2022-09-04T14:37:29.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.907 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:29.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001803495 , 2022-09-04T14:37:29.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:29.930 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:29.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:15.165445+0000", , 2022-09-04T14:37:29.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:37:29.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.287e-05 , 2022-09-04T14:37:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017164099999999999 , 2022-09-04T14:37:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.9s5 53/33 ECSubWrite(tid=66, reqid=client.4557.0:353, at_version=53'48, trim_to=0'0, roll_forward_to=48'44))", , 2022-09-04T14:37:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:29.682691+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.023272+0000", , 2022-09-04T14:37:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0547000000000002e-05 , 2022-09-04T14:37:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:30.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.560641+0000", , 2022-09-04T14:37:30.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010128000000000001 , 2022-09-04T14:37:30.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:30.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:30.615 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.615 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s1 63/33 ECSubWrite(tid=39, reqid=client.4557.0:483, at_version=63'27, trim_to=0'0, roll_forward_to=54'23))", Ignoring malparsed backtrace: 2022-09-04T14:37:30.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:37:30.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:40.953912+0000", , 2022-09-04T14:37:30.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0060000000000001e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:30.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:30.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.964245+0000", , 2022-09-04T14:37:30.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3057999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:30.699 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:37:30.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.085861+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:30.726 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:30.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.090429+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:30.757 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:30.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126736+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:37:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:37:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:30.581+0000 7f4e01c5f700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{c=[c],15=[15,14,12,11,e],1e=[1e,1b,17],2b=[27,23],43=[42]} for object 3:38890410:::smithi16339483-19:9 , 2022-09-04T14:37:30.879+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:37:30.881+0000 7fc7ed85f700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset c=[]:{c=[c,b,9]} for object 3:da2cfe71:::smithi16339483-46:7 Ignoring malparsed backtrace: 2022-09-04T14:37:31.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:31.364 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:31.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:31.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:31.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:31.393 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:31.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1233000000000002e-05 , 2022-09-04T14:37:31.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:31.417 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:31.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010974799999999999 , 2022-09-04T14:37:31.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:31.441 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:31.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.777204+0000", , 2022-09-04T14:37:31.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:31.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.778929+0000", , 2022-09-04T14:37:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9421e-05 , 2022-09-04T14:37:31.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.023165+0000", , 2022-09-04T14:37:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5854000000000001e-05 , 2022-09-04T14:37:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:37:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.909986+0000", , 2022-09-04T14:37:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:37:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910344+0000", , 2022-09-04T14:37:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000232198 Ignoring malparsed backtrace: 2022-09-04T14:37:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916659+0000", , 2022-09-04T14:37:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7067000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:31.610 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:31.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:37:31.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954801+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:31.632 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:31.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:37:31.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.081931+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:31.655 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:31.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:31.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.084401+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:31.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:31.678 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:31.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:37:31.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:32.096 INFO:tasks.ceph.ceph_manager.ceph:unique_pool_0 still trimming, waiting , 2022-09-04T14:37:32.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 is failed for ~174s , 2022-09-04T14:37:32.122+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 Ignoring malparsed backtrace: osd.3: osd_enable_op_tracker = 'true' , osd.4: osd_enable_op_tracker = 'true' , osd.5: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:37:34.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:37:34.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.753236+0000", , 2022-09-04T14:37:34.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040576400000000003 Ignoring malparsed backtrace: 2022-09-04T14:37:34.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:37:34.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.760579+0000", , 2022-09-04T14:37:34.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0071413919999999999 Ignoring malparsed backtrace: 2022-09-04T14:37:34.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:37:34.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:13.005884+0000", , 2022-09-04T14:37:34.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0337999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:34.948 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:34.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:37:34.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801792+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:34.978 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:34.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:37:34.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.815514+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:35.005 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:35.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:35.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816962+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:35.029 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:37:35.029 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:35.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:35.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:35.053 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:35.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:35.077 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:571 3.cs0 3:36aacf6f:::smithi16339483-41:head [read 0~688128,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", , 2022-09-04T14:37:35.077 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:44.949068+0000", , 2022-09-04T14:37:35.077 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 169.882957355, Ignoring malparsed backtrace: 2022-09-04T14:37:35.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.044e-06 , 2022-09-04T14:37:35.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:35.102 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011661499999999999 , 2022-09-04T14:37:35.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:35.127 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103896 , 2022-09-04T14:37:35.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:35.151 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6446000000000002e-05 , 2022-09-04T14:37:35.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:35.175 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071671+0000", , 2022-09-04T14:37:35.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:37:35.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:35.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119221+0000", , 2022-09-04T14:37:35.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1005999999999999e-05 , 2022-09-04T14:37:35.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:35.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.176076+0000", , 2022-09-04T14:37:35.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3912000000000003e-05 , 2022-09-04T14:37:35.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:35.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:35.679 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:35.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:37:35.704 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:35.705 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:35.705 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:35.747 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", , 2022-09-04T14:37:35.747 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:37:35.748 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:37:35.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1910000000000003e-06 , 2022-09-04T14:37:35.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:35.771 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.797 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011859200000000001 , 2022-09-04T14:37:35.798 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:35.798 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.841 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00043140099999999999 , 2022-09-04T14:37:35.841 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:35.841 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.864 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012459282 , 2022-09-04T14:37:35.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:35.865 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:35.888 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:35.888 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:636 3.2s1 3:402e69e6:::smithi16339483-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", , 2022-09-04T14:37:35.888 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:51.073708+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:35.911 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.069367+0000", , 2022-09-04T14:37:35.912 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 , 2022-09-04T14:37:35.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:35.934 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046877+0000", , 2022-09-04T14:37:35.935 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000108797 , 2022-09-04T14:37:35.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:35.958 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010772729999999999, , 2022-09-04T14:37:35.958 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:37:35.958 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:35.981 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:35.981 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:35.982 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:36.005 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:37:36.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.171752+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.028 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:36.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173478+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.052 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", , 2022-09-04T14:37:36.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.269101+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.206 INFO:teuthology.orchestra.run.smithi163.stdout:{ , 2022-09-04T14:37:36.206 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], , 2022-09-04T14:37:36.206 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:36.579 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:36.579 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.734051+0000", , 2022-09-04T14:37:36.579 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8319999999999995e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:36.602 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:37:36.602 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.751044+0000", , 2022-09-04T14:37:36.603 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0087759959999999995 Ignoring malparsed backtrace: 2022-09-04T14:37:36.627 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:36.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.774974+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.651 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:36.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.775016+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.676 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:37:36.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778520+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.699 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:37:36.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.780952+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.721 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:37:36.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.781059+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.745 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:37:36.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.791872+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.768 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:37:36.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.808834+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:18.787757+0000", , 2022-09-04T14:37:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 197.77050469400001, , 2022-09-04T14:37:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.079647366999999997, Ignoring malparsed backtrace: 2022-09-04T14:37:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:37:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:36.838 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:37:36.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:36.862 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:36.885 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:37:36.885 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:36.885 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:36.907 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4295e-05 , 2022-09-04T14:37:36.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:36.908 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:36.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9630000000000001e-06 , 2022-09-04T14:37:36.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:36.931 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:36.953 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2537e-05 , 2022-09-04T14:37:36.953 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:36.953 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:37.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , 2022-09-04T14:37:37.870+0000 7efdf07f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38715) UID: 0 , 2022-09-04T14:37:37.970+0000 7f84b1586700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 39181) UID: 0 Ignoring malparsed backtrace: noscrub is unset , 2022-09-04T14:37:39.991+0000 7fbf31779700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 38722) UID: 0 , 2022-09-04T14:37:39.997 INFO:teuthology.orchestra.run.smithi077.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:37:40.336 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:37:40.336 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:40.337 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:37:40.363 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.783686+0000", , 2022-09-04T14:37:40.363 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.2816e-05 , 2022-09-04T14:37:40.363 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.785480+0000", , 2022-09-04T14:37:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00075883199999999997 , 2022-09-04T14:37:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:40.413 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0066237730000000003, , 2022-09-04T14:37:40.413 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:37:40.413 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:40.439 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:37:40.440 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.789604+0000", , 2022-09-04T14:37:40.440 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.2855000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:40.465 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:40.465 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.790799+0000", , 2022-09-04T14:37:40.466 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.7930000000000003e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:40.491 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 196.490717668, , 2022-09-04T14:37:40.491 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0064107390000000004, , 2022-09-04T14:37:40.492 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:37:40.518 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:40.519 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:37:40.519 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686080+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:40.549 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:40.550 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:40.550 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686475+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:40.578 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:51.083397+0000", , 2022-09-04T14:37:40.578 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 169.22364529999999, , 2022-09-04T14:37:40.578 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0048458909999999997, Ignoring malparsed backtrace: 2022-09-04T14:37:40.604 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:40.605 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:40.605 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:37:40.629 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:40.629 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:40.629 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:37:40.651 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.2s5 104/52 ECSubWrite(tid=137, reqid=client.4557.0:937, at_version=104'31, trim_to=0'0, roll_forward_to=74'30))", , 2022-09-04T14:37:40.651 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:35:21.263375+0000", , 2022-09-04T14:37:40.651 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 139.04366698000001, Ignoring malparsed backtrace: 2022-09-04T14:37:40.673 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.7265999999999997e-05 , 2022-09-04T14:37:40.674 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:40.674 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:40.704 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.0703000000000003e-05 , 2022-09-04T14:37:40.704 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:40.705 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:41.166 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.809787+0000", , 2022-09-04T14:37:41.166 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.7349999999999999e-06 , 2022-09-04T14:37:41.167 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:41.192 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.814913+0000", , 2022-09-04T14:37:41.192 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00022232099999999999 , 2022-09-04T14:37:41.192 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:41.217 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:37:41.217 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831509+0000", , 2022-09-04T14:37:41.217 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.9999986 Ignoring malparsed backtrace: 2022-09-04T14:37:41.240 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:37:41.240 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.835569+0000", , 2022-09-04T14:37:41.240 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.8469999999999998e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:41.265 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:37:41.265 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:24.598967+0000", , 2022-09-04T14:37:41.265 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00034774300000000002 Ignoring malparsed backtrace: 2022-09-04T14:37:41.289 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.289 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:37:41.289 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.842856+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:41.312 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.313 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:37:41.313 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843682+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:41.336 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.337 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:37:41.337 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.689485+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:41.361 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.361 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.361 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:41.385 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.385 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.385 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:41.408 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.409 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.409 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:41.432 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:41.432 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.433 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:41.457 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.7007000000000004e-05 , 2022-09-04T14:37:41.457 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.457 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:41.481 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2011000000000001e-05 , 2022-09-04T14:37:41.481 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.482 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:41.514 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.105466+0000", , 2022-09-04T14:37:41.514 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:41.514 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:41.935 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.935 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.936 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:41.960 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.960 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.960 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:41.984 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:42.008 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:42.008 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:42.008 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:42.031 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4857999999999995e-05 , 2022-09-04T14:37:42.032 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:42.032 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:42.055 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00033025399999999999 , 2022-09-04T14:37:42.055 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:42.056 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:42.079 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.832850+0000", , 2022-09-04T14:37:42.079 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:42.080 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:42.103 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834407+0000", , 2022-09-04T14:37:42.103 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.5741999999999993e-05 , 2022-09-04T14:37:42.103 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:42.126 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.972557+0000", , 2022-09-04T14:37:42.126 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2726e-05 , 2022-09-04T14:37:42.127 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:42.149 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:37:42.150 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.070041+0000", , 2022-09-04T14:37:42.150 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:42.173 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:37:42.173 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.067018+0000", , 2022-09-04T14:37:42.173 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.3822e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:42.197 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:42.197 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.077233+0000", , 2022-09-04T14:37:42.197 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.605e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:42.221 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:42.221 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:37:42.221 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.083916+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:42.244 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:42.244 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:37:42.244 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:09.119233+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:42.268 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:42.268 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:42.268 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192667+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:42.380 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} , 2022-09-04T14:37:42.381 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} , 2022-09-04T14:37:42.381 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} Ignoring malparsed backtrace: 2022-09-04T14:37:44.046+0000 7faa2768e700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{2b=[2b,27,23,14],33=[33,2f],43=[43,42]} for object 3:70fb2152:::smithi16339483-29:9 , 2022-09-04T14:37:44.091+0000 7f4e208b2700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 38721) UID: 0 , 2022-09-04T14:37:44.134 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:37:45.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.052 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:45.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:45.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.076 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:45.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:45.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.105 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:45.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2022-09-04T14:37:45.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.133 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:45.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:45.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.160 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:45.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:37:45.194 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:45.194 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:45.195 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:45.219 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:45.219 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:45.220 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:45.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:37:45.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.243 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:45.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:37:45.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.267 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:45.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9420000000000007e-06 , 2022-09-04T14:37:45.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.290 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:45.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00044862 , 2022-09-04T14:37:45.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.313 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:45.336 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000782666 , 2022-09-04T14:37:45.337 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:45.337 INFO:teuthology.orchestra.run.smithi163.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:37:45.359 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.912809+0000", , 2022-09-04T14:37:45.359 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9399999999999996e-06 , 2022-09-04T14:37:45.360 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:45.382 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.682723+0000", , 2022-09-04T14:37:45.383 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00088424200000000004 , 2022-09-04T14:37:45.383 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:45.405 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.742229+0000", , 2022-09-04T14:37:45.405 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.059345506999999999 , 2022-09-04T14:37:45.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:45.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.133467+0000", , 2022-09-04T14:37:45.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020402139999999998 , 2022-09-04T14:37:45.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:45.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:45.451 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:45.451 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.2s6 104/52 ECSubWrite(tid=137, reqid=client.4557.0:937, at_version=104'31, trim_to=0'0, roll_forward_to=74'30))", Ignoring malparsed backtrace: 2022-09-04T14:37:45.863+0000 7fc7ef863700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 17=[]:{9=[9],c=[c,b],17=[17,14,12,e]} for object 3:eaf642d1:::smithi16339483-45:5 , 2022-09-04T14:37:45.873+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 , 2022-09-04T14:37:45.947 INFO:teuthology.orchestra.run.smithi163.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:37:45.971 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.792727+0000", , 2022-09-04T14:37:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000108977 , 2022-09-04T14:37:45.972 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:37:45.995 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.832814+0000", , 2022-09-04T14:37:45.996 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:37:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833086+0000", , 2022-09-04T14:37:46.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015123400000000001 Ignoring malparsed backtrace: 2022-09-04T14:37:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833223+0000", , 2022-09-04T14:37:46.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5440000000000004e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:37:46.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.834310+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:46.093 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:37:46.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.835242+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.840270+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:37:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:37:46.211 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:46.211 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:37:46.211 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:46.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 , 2022-09-04T14:37:46.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:46.234 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:46.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9475999999999999e-05 , 2022-09-04T14:37:46.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:46.256 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:46.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014419882 , 2022-09-04T14:37:46.279 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:37:46.280 INFO:teuthology.orchestra.run.smithi163.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:37:46.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:22.192220+0000", , 2022-09-04T14:37:46.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 , 2022-09-04T14:37:46.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:46.843 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.795154+0000", , 2022-09-04T14:37:46.844 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:46.875 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.802784+0000", , 2022-09-04T14:37:46.876 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5069999999999997e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:46.904 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:19.064416+0000", , 2022-09-04T14:37:46.905 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3440000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:46.940 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.940 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:37:46.941 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.780204+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:46.968 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.968 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:46.968 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.782680+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:46.991 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:46.992 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:37:46.992 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.684394+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:47.014 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:37:47.014 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:47.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:47.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:47.037 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:47.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:47.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:47.059 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:37:47.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:47.083 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:47.083 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:37:47.083 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:47.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8898999999999999e-05 , 2022-09-04T14:37:47.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:47.107 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:47.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.259e-06 , 2022-09-04T14:37:47.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:37:47.132 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:47.169 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:37:47.169 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:37:47.170 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:37:47.195 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.172799+0000", , 2022-09-04T14:37:47.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2636e-05 , 2022-09-04T14:37:47.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:47.220 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175758+0000", , 2022-09-04T14:37:47.220 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4749999999999996e-06 , 2022-09-04T14:37:47.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:47.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , 2022-09-04T14:37:47.345+0000 7ffa4ea67700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 38946) UID: 0 , 2022-09-04T14:37:47.403 INFO:teuthology.orchestra.run.smithi038.stdout:{ Ignoring malparsed backtrace: 2022-09-04T14:37:49.203+0000 7f4035433700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 , 2022-09-04T14:37:49.247+0000 7efdceb98700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 9=[]:{9=[9,8]} for object 3:55ca316a:::smithi16339483-9:5 , 2022-09-04T14:37:49.247+0000 7efdd139d700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2f=[]:{2f=[2f]} for object 3:b771c3b4:::smithi16339483-5:9 Ignoring malparsed backtrace: 2022-09-04T14:37:50.400 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:50.400 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:50.400 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:50.425 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:50.425 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:50.426 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:50.450 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:50.451 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:50.451 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:50.474 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:50.475 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:50.475 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:50.499 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4857999999999995e-05 , 2022-09-04T14:37:50.499 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:50.499 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:50.523 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00033025399999999999 , 2022-09-04T14:37:50.523 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:37:50.523 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:50.548 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.832850+0000", , 2022-09-04T14:37:50.548 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:37:50.548 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:50.573 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834407+0000", , 2022-09-04T14:37:50.573 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.5741999999999993e-05 , 2022-09-04T14:37:50.574 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:50.598 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.972557+0000", , 2022-09-04T14:37:50.598 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.2726e-05 , 2022-09-04T14:37:50.598 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:37:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.070041+0000", , 2022-09-04T14:37:50.623 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:50.647 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:37:50.648 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.067018+0000", , 2022-09-04T14:37:50.648 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.3822e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:50.672 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:50.672 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.077233+0000", , 2022-09-04T14:37:50.672 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.605e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:50.697 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:50.697 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:37:50.697 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.083916+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:50.720 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:50.721 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:37:50.721 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:09.119233+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:50.746 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:37:50.746 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:50.746 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192667+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:51.083+0000 7fc80fcb9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 39177) UID: 0 , 2022-09-04T14:37:51.086 INFO:teuthology.orchestra.run.smithi155.stdout:{ , 2022-09-04T14:37:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], Ignoring malparsed backtrace: 2022-09-04T14:37:51.254 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:51.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.254 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:51.278 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:37:51.278 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:37:51.278 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 276 Ignoring malparsed backtrace: 2022-09-04T14:37:51.303 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:37:51.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828939+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:51.328 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", , 2022-09-04T14:37:51.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.813965+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:51.352 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:37:51.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.829848+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:51.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.376 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:51.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.399 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:51.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.423 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:51.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.446 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:37:51.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.470 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.470 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:37:51.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.494 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:51.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.518 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:51.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2022-09-04T14:37:51.541 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:37:51.542 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:37:51.542 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:51.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:37:51.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.565 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:51.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0197999999999994e-05 , 2022-09-04T14:37:51.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.589 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:51.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113773 , 2022-09-04T14:37:51.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.613 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:51.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010433399999999999 , 2022-09-04T14:37:51.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:51.638 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:52.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.063 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:52.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:37:52.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.096 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:52.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:52.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.144 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:52.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:37:52.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.169 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:37:52.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:37:52.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:37:52.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.194 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:52.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3805999999999999e-05 , 2022-09-04T14:37:52.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.218 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:52.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.043749430999999998 , 2022-09-04T14:37:52.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.250 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:52.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048612599999999997 , 2022-09-04T14:37:52.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:37:52.280 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:52.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.827349+0000", , 2022-09-04T14:37:52.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:37:52.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:52.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.779801+0000", , 2022-09-04T14:37:52.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9899999999999999e-05 , 2022-09-04T14:37:52.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:52.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.780356+0000", , 2022-09-04T14:37:52.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050213600000000003 , 2022-09-04T14:37:52.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:52.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.089583+0000", , 2022-09-04T14:37:52.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7630000000000003e-06 , 2022-09-04T14:37:52.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:52.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.45252578999999998, , 2022-09-04T14:37:52.411 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:37:52.411 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:37:52.435 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 157.80642642199999, , 2022-09-04T14:37:52.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.91158610600000001, , 2022-09-04T14:37:52.435 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:37:52.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false , 2022-09-04T14:37:52.176+0000 7faa2768e700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{2b=[2b,27,23,14],33=[33,2f],43=[43,42]} for object 3:70fb2152:::smithi16339483-29:9 , 2022-09-04T14:37:52.459+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:37:53.685 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:37:53.685 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:37:53.695 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops Ignoring malparsed backtrace: osd.10: osd_enable_op_tracker = 'true' , osd.11: osd_enable_op_tracker = 'true' , osd.12: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:37:55.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.677 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:55.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:37:55.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.704 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:55.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:37:55.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:55.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.732 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:55.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5237e-05 , 2022-09-04T14:37:55.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.756 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:55.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028262799999999999 , 2022-09-04T14:37:55.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.785 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:55.810 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 588 , 2022-09-04T14:37:55.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.810 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:37:55.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.962857+0000", , 2022-09-04T14:37:55.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:55.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:55.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.856 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:55.856 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/33 ECSubWrite(tid=103, reqid=client.4557.0:622, at_version=74'71, trim_to=0'0, roll_forward_to=71'68))", Ignoring malparsed backtrace: 2022-09-04T14:37:55.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:37:55.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.058456+0000", , 2022-09-04T14:37:55.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:55.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:55.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077717+0000", , 2022-09-04T14:37:55.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0476999999999994e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:55.925 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:55.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:55.925 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:37:55.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126162+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:55.970 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:55.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:55.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.131985+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:55.993 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:55.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:55.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.199178+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:56.016 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:37:56.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.200671+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:56.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.727082+0000", , 2022-09-04T14:37:56.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9896999999999998e-05 , 2022-09-04T14:37:56.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:56.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:56.536 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.536 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:38 3.1s0 3:89c1087a:::smithi16339483-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", Ignoring malparsed backtrace: 2022-09-04T14:37:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:39 3.1s0 3:89c1087a:::smithi16339483-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", Ignoring malparsed backtrace: 2022-09-04T14:37:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:37:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.742616+0000", , 2022-09-04T14:37:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:37:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.742636+0000", , 2022-09-04T14:37:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:37:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.809273+0000", , 2022-09-04T14:37:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3797e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:56.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:37:56.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834841+0000", , 2022-09-04T14:37:56.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9700000000000005e-06 Ignoring malparsed backtrace: 2022-09-04T14:37:56.689 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:37:56.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.780066+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:56.712 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:37:56.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.909167+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:56.735 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:56.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.915797+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:56.758 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:37:56.758 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:37:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:56.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:37:56.809 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:56.809 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:56.809 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", , 2022-09-04T14:37:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:37:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:37:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4042999999999999e-05 , 2022-09-04T14:37:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, , 2022-09-04T14:37:57.239 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 , 2022-09-04T14:37:57.239 INFO:teuthology.orchestra.run.smithi038.stdout:} Ignoring malparsed backtrace: 2022-09-04T14:37:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.773854+0000", , 2022-09-04T14:37:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682855+0000", , 2022-09-04T14:37:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028397999999999997 , 2022-09-04T14:37:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.683179+0000", , 2022-09-04T14:37:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3288000000000001e-05 , 2022-09-04T14:37:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:37:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.157225+0000", , 2022-09-04T14:37:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:37:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:37:57.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.909359+0000", , 2022-09-04T14:37:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9675999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:37:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.911005+0000", , 2022-09-04T14:37:57.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017453799999999999 Ignoring malparsed backtrace: 2022-09-04T14:37:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:37:57.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.962438+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:57.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:57.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.078090+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:57.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:57.636 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:57.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:57.668 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:57.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:57.696 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:37:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:57.722 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.999999 , 2022-09-04T14:37:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:57.749 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.844e-06 , 2022-09-04T14:37:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:57.775 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:37:58.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783268+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:58.372 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:58.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:37:58.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.784192+0000", Ignoring malparsed backtrace: 2022-09-04T14:37:58.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:58.409 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:58.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:58.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:58.458 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:58.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:37:58.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:58.491 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:37:58.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:37:58.515 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:58.515 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:37:58.516 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:58.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967296 , 2022-09-04T14:37:58.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:58.540 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1287000000000001e-05 , 2022-09-04T14:37:58.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:37:58.563 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:37:58.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026947641000000001 , 2022-09-04T14:37:58.589 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:37:58.589 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:37:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916291+0000", , 2022-09-04T14:37:58.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999967 , 2022-09-04T14:37:58.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:58.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954752+0000", , 2022-09-04T14:37:58.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3892e-05 , 2022-09-04T14:37:58.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:37:58.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.961457+0000", , 2022-09-04T14:37:58.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063470080000000003 , 2022-09-04T14:37:58.659 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:37:58.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:37:58.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.084038+0000", , 2022-09-04T14:37:58.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:37:58.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:37:58.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126727+0000", , 2022-09-04T14:37:58.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2199e-05 Ignoring malparsed backtrace: 2022-09-04T14:37:58.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:37:58.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.136037+0000", , 2022-09-04T14:37:58.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031234890000000001 Ignoring malparsed backtrace: osd.8: osd_enable_op_tracker = 'false' , osd.9: osd_enable_op_tracker = 'false' , osd.10: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:38:01.057 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:38:01.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.721955+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.722263+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.733771+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.762387+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.761146+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:01.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.241 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.241 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:01.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.271 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:01.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.301 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:01.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:01.326 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/33 ECSubWrite(tid=167, reqid=client.4557.0:127, at_version=36'5, trim_to=0'0, roll_forward_to=36'4))", , 2022-09-04T14:38:01.327 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.827349+0000", , 2022-09-04T14:38:01.327 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 228.220512945, Ignoring malparsed backtrace: 2022-09-04T14:38:01.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999981 , 2022-09-04T14:38:01.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.354 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:01.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0049e-05 , 2022-09-04T14:38:01.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.379 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:01.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035543700000000001 , 2022-09-04T14:38:01.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.408 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:01.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098362300000000009 , 2022-09-04T14:38:01.454 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:01.454 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:01.495 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.584984+0000", , 2022-09-04T14:38:01.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000224804 , 2022-09-04T14:38:01.496 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:01.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:38:01.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:15.193104+0000", , 2022-09-04T14:38:01.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.676e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:01.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8930000000000004e-06 , 2022-09-04T14:38:01.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:01.987 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:02.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4360000000000004e-06 , 2022-09-04T14:38:02.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.012 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:02.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4911000000000002e-05 , 2022-09-04T14:38:02.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.036 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:02.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6950999999999997e-05 , 2022-09-04T14:38:02.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.070 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.186670524 , 2022-09-04T14:38:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00030160799999999998 , 2022-09-04T14:38:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:02.148 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 159 , 2022-09-04T14:38:02.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.148 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:38:02.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858465+0000", , 2022-09-04T14:38:02.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:02.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783207+0000", , 2022-09-04T14:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6526000000000001e-05 , 2022-09-04T14:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783968+0000", , 2022-09-04T14:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4161999999999999e-05 , 2022-09-04T14:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784796+0000", , 2022-09-04T14:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:02.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:02.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801903+0000", , 2022-09-04T14:38:02.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0795999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:02.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:38:02.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833699+0000", , 2022-09-04T14:38:02.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000470352 Ignoring malparsed backtrace: 2022-09-04T14:38:02.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:38:02.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.909850+0000", , 2022-09-04T14:38:02.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00047020100000000001 Ignoring malparsed backtrace: 2022-09-04T14:38:02.362 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:02.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:38:02.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.173569+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:02.388 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:02.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:02.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:21.263683+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:02.414 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:02.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:02.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:22.192339+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:02.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.891 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:02.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:02.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.914 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:02.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:02.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.937 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:02.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:38:02.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.961 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:02.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", Ignoring malparsed backtrace: 2022-09-04T14:38:02.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:02.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:02.986 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:03.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5529999999999993e-06 , 2022-09-04T14:38:03.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:03.009 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:03.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6615999999999999e-05 , 2022-09-04T14:38:03.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:03.040 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:03.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3522e-05 , 2022-09-04T14:38:03.062 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:03.063 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:03.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949157+0000", , 2022-09-04T14:38:03.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011005699999999999 , 2022-09-04T14:38:03.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:03.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0080371309999999994, , 2022-09-04T14:38:03.108 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:38:03.108 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", Ignoring malparsed backtrace: 2022-09-04T14:38:03.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:38:03.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.951133+0000", , 2022-09-04T14:38:03.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9339999999999995e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:03.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:38:03.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.951159+0000", , 2022-09-04T14:38:03.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.822e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:03.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:38:03.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069430+0000", , 2022-09-04T14:38:03.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0192e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:03.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:38:03.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.072005+0000", , 2022-09-04T14:38:03.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080663100000000004 Ignoring malparsed backtrace: 2022-09-04T14:38:03.244 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:03.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:38:03.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119125+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:03.271 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:03.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:38:03.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.175762+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:15.747298+0000", , 2022-09-04T14:38:03.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:38:03.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:03.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.779865+0000", , 2022-09-04T14:38:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4969999999999998e-06 , 2022-09-04T14:38:03.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843414+0000", , 2022-09-04T14:38:03.798 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023020380000000002 , 2022-09-04T14:38:03.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:03.838 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.847758+0000", , 2022-09-04T14:38:03.839 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0035534299999999998 , 2022-09-04T14:38:03.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 348 , 2022-09-04T14:38:03.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:38:03.888 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680034+0000", , 2022-09-04T14:38:03.889 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:03.915 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:38:03.916 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680080+0000", , 2022-09-04T14:38:03.916 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5889999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:03.939 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:38:03.939 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.073689+0000", , 2022-09-04T14:38:03.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8619999999999992e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:03.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", , 2022-09-04T14:38:03.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:51.074228+0000", , 2022-09-04T14:38:03.963 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8622999999999996e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:03.986 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:38:03.986 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.080006+0000", , 2022-09-04T14:38:03.986 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010202002999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:04.010 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:04.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:38:04.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046784+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:04.034 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:59.052261+0000", , 2022-09-04T14:38:04.034 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 184.65473155999999, , 2022-09-04T14:38:04.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013656681, Ignoring malparsed backtrace: 2022-09-04T14:38:04.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:04.057 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:04.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting_for_map not empty", Ignoring malparsed backtrace: 2022-09-04T14:38:04.082 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs1 99/52 ECSubWrite(tid=119, reqid=client.4557.0:916, at_version=99'71, trim_to=0'0, roll_forward_to=89'67))", , 2022-09-04T14:38:04.083 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:16.173164+0000", , 2022-09-04T14:38:04.083 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 167.53382830000001, Ignoring malparsed backtrace: 2022-09-04T14:38:04.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 , 2022-09-04T14:38:04.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:04.108 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:04.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3764999999999999e-05 , 2022-09-04T14:38:04.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:04.135 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: osd.3: osd_enable_op_tracker = 'false' , osd.4: osd_enable_op_tracker = 'false' , osd.5: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:38:07.381 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} , 2022-09-04T14:38:07.391+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 , 2022-09-04T14:38:07.400 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:38:07.921 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.781832+0000", , 2022-09-04T14:38:07.922 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00011127400000000001 , 2022-09-04T14:38:07.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:07.948 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:07.948 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:07.948 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.2s5 42/41 ECSubWrite(tid=12, reqid=client.4557.0:204, at_version=42'9, trim_to=0'0, roll_forward_to=0'0))", Ignoring malparsed backtrace: 2022-09-04T14:38:07.974 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:38:07.974 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.784593+0000", , 2022-09-04T14:38:07.975 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.8321999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:07.999 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:07.999 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.784948+0000", , 2022-09-04T14:38:07.999 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.4426999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:08.024 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:38:08.024 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:08.025 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:38:08.049 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.790605+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:08.075 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:08.075 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:08.075 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.807804+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:08.099 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:38:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:38:08.100 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:08.124 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:08.124 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:08.124 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:38:08.150 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:08.150 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:08.151 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:38:08.184 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:38:08.184 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:38:08.184 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:08.211 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967296 , 2022-09-04T14:38:08.211 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:08.212 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:08.239 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.3780000000000001e-06 , 2022-09-04T14:38:08.239 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:08.239 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:08.269 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.003712117 , 2022-09-04T14:38:08.270 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:38:08.270 INFO:teuthology.orchestra.run.smithi077.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:08.296 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194301+0000", , 2022-09-04T14:38:08.296 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.999999 , 2022-09-04T14:38:08.297 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:08.763 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.773838+0000", , 2022-09-04T14:38:08.763 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:38:08.763 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:08.794 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.814438+0000", , 2022-09-04T14:38:08.794 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.5865999999999998e-05 , 2022-09-04T14:38:08.795 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:08.826 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.831916+0000", , 2022-09-04T14:38:08.826 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.6765e-05 , 2022-09-04T14:38:08.827 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:08.851 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:38:08.852 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834389+0000", , 2022-09-04T14:38:08.852 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:08.876 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:38:08.876 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:24.598439+0000", , 2022-09-04T14:38:08.877 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.8315999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:08.901 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:08.902 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.844031+0000", , 2022-09-04T14:38:08.902 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.4280000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:08.927 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:08.927 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:38:08.927 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843585+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:08.952 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:08.952 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:38:08.953 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684729+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:08.977 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:08.977 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:08.977 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684935+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:09.002 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.002 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.002 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:09.027 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.027 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.027 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:09.052 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.052 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.052 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:09.077 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:38:09.077 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.077 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:09.101 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 9.5079999999999994e-06 , 2022-09-04T14:38:09.102 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.102 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:09.125 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.4711e-05 , 2022-09-04T14:38:09.125 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.125 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:09.279 INFO:teuthology.orchestra.run.smithi077.stdout:{ , 2022-09-04T14:38:09.279 INFO:teuthology.orchestra.run.smithi077.stdout: "ops": [], , 2022-09-04T14:38:09.280 INFO:teuthology.orchestra.run.smithi077.stdout: "num_ops": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:09.571 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:38:09.571 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.571 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:38:09.594 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.594 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.594 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:09.616 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.616 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.617 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:38:09.643 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:38:09.644 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:38:09.644 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:09.675 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00016793999999999999 , 2022-09-04T14:38:09.676 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.676 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:09.698 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.4440000000000002e-06 , 2022-09-04T14:38:09.698 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:09.699 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:09.721 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:38:09.721 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:38:09.721 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:38:09.744 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971465+0000", , 2022-09-04T14:38:09.744 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.6743000000000001e-05 , 2022-09-04T14:38:09.745 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:09.768 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973441+0000", , 2022-09-04T14:38:09.768 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.5250000000000002e-06 , 2022-09-04T14:38:09.769 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:09.805 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0062572299999999999, , 2022-09-04T14:38:09.805 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:38:09.805 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:38:09.829 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:38:09.829 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.075933+0000", , 2022-09-04T14:38:09.829 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.5437e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:09.853 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:09.853 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.082100+0000", , 2022-09-04T14:38:09.853 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.0219999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:09.877 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 180.42954484500001, , 2022-09-04T14:38:09.877 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0050833759999999997, , 2022-09-04T14:38:09.877 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:09.901 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.901 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:38:09.901 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192347+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:09.925 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:09.925 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:09.926 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194517+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:10.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:10.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.721957+0000", , 2022-09-04T14:38:10.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.872e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:38:10.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.725544+0000", , 2022-09-04T14:38:10.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2323000000000004e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:10.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:38:10.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.736115+0000", , 2022-09-04T14:38:10.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5586000000000004e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:10.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:38:10.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.786188+0000", , 2022-09-04T14:38:10.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.023673610000000001 Ignoring malparsed backtrace: 2022-09-04T14:38:10.934 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:10.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:10.934 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:10.957 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:10.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:38:10.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.759698+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:10.981 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:10.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:38:10.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.787168+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:38:11.004 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:11.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:38:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:11.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:11.051 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:11.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:14.246843+0000", , 2022-09-04T14:38:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:11.070+0000 7f793eff5700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{17=[17,14,12,e,c],27=[27,25,23,22,1f],43=[42]} for object 3:a9d45bed:::smithi16339483-27:2 , 2022-09-04T14:38:11.144+0000 7f7942ffd700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{17=[17,14,12,e,c],27=[27,25,23,22,1f],43=[42]} for object 3:a9d45bed:::smithi16339483-27:2 , 2022-09-04T14:38:11.198 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} Ignoring malparsed backtrace: 2022-09-04T14:38:13.152+0000 7f4e01c5f700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{c=[c],15=[15,14,12,11,e],1e=[1e,1b,17],2b=[27,23],43=[42]} for object 3:38890410:::smithi16339483-19:9 , 2022-09-04T14:38:13.152+0000 7f8152796700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset e=[]:{e=[e,c,b,9]} for object 3:0dcdd893:::smithi16339483-22:8 , 2022-09-04T14:38:13.152+0000 7efdd139d700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 2f=[]:{2f=[2f]} for object 3:b771c3b4:::smithi16339483-5:9 Ignoring malparsed backtrace: 2022-09-04T14:38:14.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.800364+0000", , 2022-09-04T14:38:14.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050021950000000001 , 2022-09-04T14:38:14.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:14.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.803062+0000", , 2022-09-04T14:38:14.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0570000000000004e-06 , 2022-09-04T14:38:14.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:14.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0064098310000000004, , 2022-09-04T14:38:14.062 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:38:14.062 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:38:14.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:38:14.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.780283+0000", , 2022-09-04T14:38:14.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5169e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:14.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:14.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.782821+0000", , 2022-09-04T14:38:14.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8949999999999998e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:14.138 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 224.30810684799999, , 2022-09-04T14:38:14.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0083674140000000001, , 2022-09-04T14:38:14.139 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:38:14.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.694554+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:14.191 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:14.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.733607+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:41.959652+0000", , 2022-09-04T14:38:14.220 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 212.031780846, , 2022-09-04T14:38:14.221 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058627230000000002, Ignoring malparsed backtrace: 2022-09-04T14:38:14.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:14.249 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:14.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:38:14.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:14.273 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:14.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:14.295 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs2 99/52 ECSubWrite(tid=117, reqid=client.4557.0:914, at_version=99'69, trim_to=0'0, roll_forward_to=89'67))", , 2022-09-04T14:38:14.296 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:16.170517+0000", , 2022-09-04T14:38:14.296 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 177.82091586199999, Ignoring malparsed backtrace: 2022-09-04T14:38:14.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7110000000000001e-05 , 2022-09-04T14:38:14.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:14.319 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:14.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012678799999999999 , 2022-09-04T14:38:14.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:14.343 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:14.367 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:14.367 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.0s5 100/33 ECSubWrite(tid=135, reqid=client.4557.0:919, at_version=100'73, trim_to=0'0, roll_forward_to=86'71))", , 2022-09-04T14:38:14.367 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:17.271465+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:14.711 INFO:teuthology.orchestra.run.smithi038.stdout:{ , 2022-09-04T14:38:14.711 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], , 2022-09-04T14:38:14.711 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, Ignoring malparsed backtrace: 2022-09-04T14:38:14.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:38:14.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.818244+0000", , 2022-09-04T14:38:14.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:14.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:14.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.861405+0000", , 2022-09-04T14:38:14.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7317e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:14.940 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:14.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:38:14.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682473+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:14.966 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:14.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:14.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.741502+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:14.991 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:14.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:14.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.911028+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:15.016 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:38:15.017 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:15.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:38:15.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:15.041 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:15.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:15.067 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/33 ECSubWrite(tid=103, reqid=client.4557.0:622, at_version=74'71, trim_to=0'0, roll_forward_to=71'68))", , 2022-09-04T14:38:15.067 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:50.983613+0000", , 2022-09-04T14:38:15.067 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 203.873622888, Ignoring malparsed backtrace: 2022-09-04T14:38:15.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 , 2022-09-04T14:38:15.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:15.092 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:15.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0476999999999994e-05 , 2022-09-04T14:38:15.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:15.125 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:15.151 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:15.152 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.bs1 92/33 ECSubWrite(tid=261, reqid=client.4557.0:859, at_version=92'50, trim_to=0'0, roll_forward_to=90'48))", , 2022-09-04T14:38:15.152 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:35:09.126034+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:15.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126162+0000", , 2022-09-04T14:38:15.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0876000000000001e-05 , 2022-09-04T14:38:15.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:15.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.131985+0000", , 2022-09-04T14:38:15.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238249 , 2022-09-04T14:38:15.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.199178+0000", , 2022-09-04T14:38:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2925000000000001e-05 , 2022-09-04T14:38:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.200671+0000", , 2022-09-04T14:38:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001318077 , 2022-09-04T14:38:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9896999999999998e-05 , 2022-09-04T14:38:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:38 3.1s0 3:89c1087a:::smithi16339483-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:38:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:12.728511+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:15.845 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:15.845 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:39 3.1s0 3:89c1087a:::smithi16339483-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:38:15.845 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:12.728534+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:15.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.742616+0000", , 2022-09-04T14:38:15.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:38:15.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:15.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.742636+0000", , 2022-09-04T14:38:15.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:38:15.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:15.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.809273+0000", , 2022-09-04T14:38:15.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3797e-05 , 2022-09-04T14:38:15.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:15.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834841+0000", , 2022-09-04T14:38:15.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9700000000000005e-06 , 2022-09-04T14:38:15.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:16.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:38:16.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.780066+0000", , 2022-09-04T14:38:16.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:16.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:16.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.909167+0000", , 2022-09-04T14:38:16.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7766000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:16.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:16.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.915797+0000", , 2022-09-04T14:38:16.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1063000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.976981+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:16.112 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:16.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:38:16.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:59.074326+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:16.137 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:38:16.137 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:16.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:38:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", , 2022-09-04T14:38:16.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:38:16.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:16.186 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:16.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:16.414 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} , 2022-09-04T14:38:16.437 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} , 2022-09-04T14:38:16.461 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} Ignoring malparsed backtrace: 2022-09-04T14:38:18.341 INFO:teuthology.orchestra.run.smithi077.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:38:18.342 INFO:teuthology.orchestra.run.smithi077.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:38:18.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops Ignoring malparsed backtrace: 2022-09-04T14:38:19.853 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ , 2022-09-04T14:38:19.854 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:19.854 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:8 3.5s0 3:a83f9681:::smithi16339483-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", Ignoring malparsed backtrace: 2022-09-04T14:38:19.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:19.883 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:19.884 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:9 3.5s0 3:a83f9681:::smithi16339483-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", Ignoring malparsed backtrace: 2022-09-04T14:38:19.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:19.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.765917+0000", , 2022-09-04T14:38:19.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:19.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:19.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.765934+0000", , 2022-09-04T14:38:19.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:19.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:38:19.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.789258+0000", , 2022-09-04T14:38:19.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9750000000000004e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:20.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:38:20.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.790348+0000", , 2022-09-04T14:38:20.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104929 Ignoring malparsed backtrace: 2022-09-04T14:38:20.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:38:20.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.790444+0000", , 2022-09-04T14:38:20.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4121000000000005e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:20.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:38:20.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.866072+0000", , 2022-09-04T14:38:20.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010347400000000001 Ignoring malparsed backtrace: 2022-09-04T14:38:20.082 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:20.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.082 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:20.105 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:20.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783695+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:20.129 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:20.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784730+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:20.152 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:38:20.152 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:20.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:20.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.201 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:20.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.228 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:38:20.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.254 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:20.276 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:20.276 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:38:20.277 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:20.601 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, , 2022-09-04T14:38:20.602 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 , 2022-09-04T14:38:20.602 INFO:teuthology.orchestra.run.smithi155.stdout:} Ignoring malparsed backtrace: 2022-09-04T14:38:20.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.806 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:38:20.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.830 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:38:20.830 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:20.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:20.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.853 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:20.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.838e-05 , 2022-09-04T14:38:20.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.875 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:20.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1827e-05 , 2022-09-04T14:38:20.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.897 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:20.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.841089+0000", , 2022-09-04T14:38:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:20.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.847619+0000", , 2022-09-04T14:38:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2782999999999999e-05 , 2022-09-04T14:38:20.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:20.966 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:570 3.cs0 3:36aacf6f:::smithi16339483-41:head [read 0~688128,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", Ignoring malparsed backtrace: 2022-09-04T14:38:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:20.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949072+0000", , 2022-09-04T14:38:20.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7199999999999997e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:21.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:38:21.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.954373+0000", , 2022-09-04T14:38:21.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013589299999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:21.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:38:21.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.957384+0000", , 2022-09-04T14:38:21.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024630870000000001 Ignoring malparsed backtrace: 2022-09-04T14:38:21.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:38:21.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.957504+0000", , 2022-09-04T14:38:21.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025825209999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:21.079 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:21.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:21.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.070060+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:21.102 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:21.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:21.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071764+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:21.125 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:21.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:21.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119872+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:21.347+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:38:21.423 INFO:teuthology.orchestra.run.smithi163.stdout:{ , 2022-09-04T14:38:21.424 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], Ignoring malparsed backtrace: 2022-09-04T14:38:21.623 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:38:21.624 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:21.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:38:21.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:21.647 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:21.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:21.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:21.670 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:21.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:38:21.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:21.694 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:21.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:21.716 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:38:21.717 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:38:21.717 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:21.741 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:38:21.741 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:38:21.741 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:38:21.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:38:21.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:21.764 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:21.787 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:38:21.787 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:21.787 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:21.809 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3723999999999999e-05 , 2022-09-04T14:38:21.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:21.810 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:21.833 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:21.833 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:705 3.0s0 3:0adb9d60:::smithi16339483-39:23 [read 0~3211264,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", , 2022-09-04T14:38:21.833 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:59.046784+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:21.855 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.060599+0000", , 2022-09-04T14:38:21.856 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2547999999999999e-05 , 2022-09-04T14:38:21.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:21.878 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:38:21.879 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.116537+0000", , 2022-09-04T14:38:21.879 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:38:21.901 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:21.902 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.172062+0000", , 2022-09-04T14:38:21.902 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011780700000000001 Ignoring malparsed backtrace: 2022-09-04T14:38:21.929 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 180.35156061500001, , 2022-09-04T14:38:21.930 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024978565000000001, , 2022-09-04T14:38:21.930 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:21.952 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:21.952 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:38:21.953 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.247412+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:22.369 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} , 2022-09-04T14:38:22.374+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 , 2022-09-04T14:38:22.390 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} Ignoring malparsed backtrace: 2022-09-04T14:38:24.499+0000 7fbf31779700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 38722) UID: 0 , 2022-09-04T14:38:24.600+0000 7fc80fcb9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 39177) UID: 0 , 2022-09-04T14:38:24.613 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:38:25.315 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:25.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:38:25.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783268+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:25.342 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:25.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:25.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.784192+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:25.366 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:38:25.366 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:25.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:25.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:25.391 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:25.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:38:25.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:25.431 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:25.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:38:25.463 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:25.464 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:38:25.464 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:25.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967296 , 2022-09-04T14:38:25.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:25.495 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:25.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1287000000000001e-05 , 2022-09-04T14:38:25.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:25.522 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:25.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026947641000000001 , 2022-09-04T14:38:25.552 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:25.553 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:25.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916291+0000", , 2022-09-04T14:38:25.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999967 , 2022-09-04T14:38:25.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:25.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954752+0000", , 2022-09-04T14:38:25.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3892e-05 , 2022-09-04T14:38:25.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:25.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.961457+0000", , 2022-09-04T14:38:25.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063470080000000003 , 2022-09-04T14:38:25.638 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:38:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.084038+0000", , 2022-09-04T14:38:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.999999 Ignoring malparsed backtrace: 2022-09-04T14:38:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:38:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126727+0000", , 2022-09-04T14:38:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2199e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:38:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.136037+0000", , 2022-09-04T14:38:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031234890000000001 Ignoring malparsed backtrace: 2022-09-04T14:38:26.027 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops , 2022-09-04T14:38:26.042+0000 7fdfc7e2d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 39182) UID: 0 , 2022-09-04T14:38:26.144+0000 7f84b1586700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 39181) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:38:26.191 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:26.192 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:38:26.192 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.783075+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:26.216 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:26.216 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:38:26.217 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.783728+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:26.240 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:26.240 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:38:26.240 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.794314+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:26.263 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:26.264 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:26.264 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:26.287 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:26.287 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:26.287 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:38:26.311 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:26.311 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:26.311 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:38:26.334 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:38:26.335 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:26.335 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:26.359 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00019173399999999999 , 2022-09-04T14:38:26.360 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:26.360 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:26.382 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.7604999999999998e-05 , 2022-09-04T14:38:26.383 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:26.383 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:26.406 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:51.083397+0000", , 2022-09-04T14:38:26.406 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:38:26.406 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:26.429 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.069543+0000", , 2022-09-04T14:38:26.429 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.7461999999999998e-05 , 2022-09-04T14:38:26.429 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:26.452 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.073064+0000", , 2022-09-04T14:38:26.452 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.1814000000000001e-05 , 2022-09-04T14:38:26.453 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:26.476 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:38:26.476 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:21.263375+0000", , 2022-09-04T14:38:26.476 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:26.502 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:38:26.502 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192425+0000", , 2022-09-04T14:38:26.502 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.6873999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:26.539 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:26.539 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194589+0000", , 2022-09-04T14:38:26.540 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.3243e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:27.015 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:12.809935+0000", , 2022-09-04T14:38:27.015 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.5699999999999999e-05 , 2022-09-04T14:38:27.016 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:27.038 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:38:27.038 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.819224+0000", , 2022-09-04T14:38:27.038 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:27.061 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:38:27.062 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834511+0000", , 2022-09-04T14:38:27.062 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.1317999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:27.084 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:27.084 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.836705+0000", , 2022-09-04T14:38:27.084 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000211332 Ignoring malparsed backtrace: 2022-09-04T14:38:27.106 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:27.107 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:38:27.107 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.842242+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:27.129 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:27.129 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:38:27.129 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843087+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:27.151 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:27.151 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:27.152 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.844135+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:27.174 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:27.175 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:27.175 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:27.198 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:27.198 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:27.198 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:27.220 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:27.220 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:27.220 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:27.242 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:38:27.243 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:27.243 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:27.265 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.2359999999999998e-06 , 2022-09-04T14:38:27.265 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:27.265 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:27.290 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 9.7752999999999997e-05 , 2022-09-04T14:38:27.291 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:27.291 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:27.314 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.103422+0000", , 2022-09-04T14:38:27.314 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:38:27.314 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:27.339 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.105511+0000", , 2022-09-04T14:38:27.340 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.6399999999999997e-06 , 2022-09-04T14:38:27.340 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 , 2022-09-04T14:38:27.857+0000 7f7962451700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 38719) UID: 0 , 2022-09-04T14:38:27.866 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring Ignoring malparsed backtrace: 2022-09-04T14:38:30.249 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub , 2022-09-04T14:38:30.263 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} , 2022-09-04T14:38:30.286 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} Ignoring malparsed backtrace: 2022-09-04T14:38:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.734459+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:12.754546+0000", , 2022-09-04T14:38:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 257.72154897500002, , 2022-09-04T14:38:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013682362999999999, Ignoring malparsed backtrace: 2022-09-04T14:38:30.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.556 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.556 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:30.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.581 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.581 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:30.605 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.605 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.605 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:30.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.630 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:30.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.655 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:30.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.686 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:30.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.732 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:38:30.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.768 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:38:30.768 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:30.792 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:38:30.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:30.792 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:30.817 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:30.817 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.cs4 47/33 ECSubWrite(tid=21, reqid=client.4557.0:312, at_version=47'15, trim_to=0'0, roll_forward_to=45'11))", , 2022-09-04T14:38:30.817 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.843561+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:30.849 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680147+0000", , 2022-09-04T14:38:30.849 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 , 2022-09-04T14:38:30.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:30.878 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.680197+0000", , 2022-09-04T14:38:30.878 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3619999999999997e-06 , 2022-09-04T14:38:30.879 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:30.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.116573+0000", , 2022-09-04T14:38:30.906 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8956999999999998e-05 , 2022-09-04T14:38:30.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:30.931 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.116638+0000", , 2022-09-04T14:38:30.931 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6970000000000002e-05 , 2022-09-04T14:38:30.931 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:30.954 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.264206+0000", , 2022-09-04T14:38:30.955 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2010000000000005e-06 , 2022-09-04T14:38:30.955 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:31.422 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7390000000000005e-06 , 2022-09-04T14:38:31.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:31.423 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:31.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5979999999999999e-06 , 2022-09-04T14:38:31.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:31.458 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:31.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.779e-06 , 2022-09-04T14:38:31.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:31.492 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:31.524 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:38:31.524 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:38:31.524 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:38:31.549 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833787+0000", , 2022-09-04T14:38:31.549 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5868e-05 , 2022-09-04T14:38:31.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:31.574 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.834525+0000", , 2022-09-04T14:38:31.574 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3190000000000002e-06 , 2022-09-04T14:38:31.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:31.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0071120929999999999, , 2022-09-04T14:38:31.598 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:38:31.598 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:38:31.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:38:31.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.088704+0000", , 2022-09-04T14:38:31.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6015e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:31.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:31.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.090547+0000", , 2022-09-04T14:38:31.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0140000000000001e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:31.688 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 212.34480385099999, , 2022-09-04T14:38:31.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079376229999999996, , 2022-09-04T14:38:31.688 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:31.712 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:31.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:38:31.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.226201+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:31.737 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:31.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:38:31.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.171105+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:31.761 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:31.761 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:31.761 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173487+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:31.785 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:31.785 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:31.786 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:31.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:31.809 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:31.809 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:31.833 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:31.833 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:31.833 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:32.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6190000000000006e-06 , 2022-09-04T14:38:32.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:32.303 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:32.328 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:38:32.328 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:38:32.329 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:38:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.809465+0000", , 2022-09-04T14:38:32.358 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3323e-05 , 2022-09-04T14:38:32.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.780521+0000", , 2022-09-04T14:38:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7069999999999998e-06 , 2022-09-04T14:38:32.391 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082715489999999996, , 2022-09-04T14:38:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:38:32.414 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:38:32.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:38:32.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683503+0000", , 2022-09-04T14:38:32.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015109699999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:32.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:32.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.695351+0000", , 2022-09-04T14:38:32.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00054069899999999995 Ignoring malparsed backtrace: 2022-09-04T14:38:32.483 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 230.32616022100001, , 2022-09-04T14:38:32.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0082206699999999994, , 2022-09-04T14:38:32.484 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:32.507 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:32.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:38:32.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:41.959946+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:32.529 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:32.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:32.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:45.964130+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:32.552 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:58.086709+0000", , 2022-09-04T14:38:32.553 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 214.196586158, , 2022-09-04T14:38:32.553 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057242150000000004, Ignoring malparsed backtrace: 2022-09-04T14:38:32.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:32.576 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:32.576 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:38:32.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:32.599 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:32.599 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:32.622 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs2 99/52 ECSubWrite(tid=120, reqid=client.4557.0:916, at_version=0'0, trim_to=0'0, roll_forward_to=99'71))", , 2022-09-04T14:38:32.622 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:16.175461+0000", , 2022-09-04T14:38:32.623 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 196.10783394000001, Ignoring malparsed backtrace: 2022-09-04T14:38:32.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00028758599999999999 , 2022-09-04T14:38:32.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:32.646 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:33.145 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:33.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:38:33.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:21.828640+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:33.182 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:33.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:33.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.831718+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:33.213 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:23.862141+0000", , 2022-09-04T14:38:33.213 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 249.26385394600001, , 2022-09-04T14:38:33.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.44948170599999998, Ignoring malparsed backtrace: 2022-09-04T14:38:33.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:33.239 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:33.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:33.261 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s5 63/33 ECSubWrite(tid=38, reqid=client.4557.0:482, at_version=63'26, trim_to=0'0, roll_forward_to=54'23))", , 2022-09-04T14:38:33.262 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:39.910372+0000", , 2022-09-04T14:38:33.262 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 233.21562273800001, Ignoring malparsed backtrace: 2022-09-04T14:38:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0542000000000001e-05 , 2022-09-04T14:38:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:33.284 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:33.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000221738 , 2022-09-04T14:38:33.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:33.307 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:33.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011815 , 2022-09-04T14:38:33.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:33.329 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:33.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.057561+0000", , 2022-09-04T14:38:33.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4196999999999997e-05 , 2022-09-04T14:38:33.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:33.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.121672+0000", , 2022-09-04T14:38:33.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4676e-05 , 2022-09-04T14:38:33.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:33.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:38:33.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077956+0000", , 2022-09-04T14:38:33.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:33.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:33.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126271+0000", , 2022-09-04T14:38:33.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6128999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:33.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:33.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126610+0000", , 2022-09-04T14:38:33.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8765999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:33.470 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:33.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:38:33.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.187265+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:33.494 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:33.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:38:33.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.187311+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:33.917+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 , 2022-09-04T14:38:33.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} , 2022-09-04T14:38:33.977 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} Ignoring malparsed backtrace: 2022-09-04T14:38:35.508 INFO:teuthology.orchestra.run.smithi077.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:38:35.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight , 2022-09-04T14:38:35.535+0000 7f816f3e5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 38936) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:38:36.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:36.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.827269+0000", , 2022-09-04T14:38:36.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6792000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:38:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.829560+0000", , 2022-09-04T14:38:36.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052511799999999998 Ignoring malparsed backtrace: 2022-09-04T14:38:36.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:36.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.830324+0000", , 2022-09-04T14:38:36.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4219999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:36.898 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 252.969418166, , 2022-09-04T14:38:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.015539999000000001, , 2022-09-04T14:38:36.899 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 252.953493151, , 2022-09-04T14:38:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0083750179999999997, , 2022-09-04T14:38:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.830207+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:37.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.127 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:37.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.150 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:37.174 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:710 3.fs0 3:fcedbc97:::smithi16339483-12:head [create,setxattr `UYtaSad_Q]SlpZRUVp[^mgVZWskpd`hpTq[ZsaR (6244) in=40b,setxattr W[YVOP^ZcVnbps\\qtRfsa\\ (454) in=22b,setxattr bbSSoRd^nekRUpoQbpOaqNXaQ[cjNqZk`VQ[Uij^ (1681) in=40b,setxattr bhnkUap\\]Q\\]Qlj`qg]UUX^WN^qamSYWXZ^VRYrV (842) in=40b,setxattr QSqa_]a (454) in=7b,setxattr ehd`SWb^csbaeOcU]rR\\om\\rn`_[_djSr\\YaqsPp (19224) in=40b,setxattr k`Zt^^Sa_jpTnSng[aVgQSPnrsmp]VWXkZlQVXVh (807) in=40b,setxattr ]bf]QQYlbfUrdb[gX[RhYVWe]VPYcbcZil^qOdda (1577) in=40b,setxattr VlbXQQVWYUY_ZTetaobXXgS]\\Y_ehTc^plm`Tme` (6944) in=40b,setxattr kra_eRqj^t\\^cRSOYYbiQo_\\rgWOdaYVsfmo]s^k (105) in=40b,setxattr ZOas\\esOqeaW\\`Sfq_\\idRPNeg^khstP]mZhUiNj (1578) in=40b,setxattr YNcj\\gTfg\\bYTftmcerirR^ZslsUfVQ\\^f\\beeSg (242) in=40b,setxattr eTbspWNNOQQlabghZahOnrUYOfr`SUmadPoNUpPV (1611) in=40b,setxattr lgX`pe[nkhXbs[k^Oiplqij^e^[p\\qbjijWkZmgp (243) in=40b,setxattr hPXadn\\XiWq^aejWaiQprNR]aZdNSoWen[bmP^di (19716) in=40b,setxattr gPmddOhoP`ZcYTtZjZgiqVSjlc^`SW\\_pUNUpa\\^ (701) in=40b,setxattr NNg\\]V\\SbcXtrROXgNrn[kbQdfRmfTba\\dWNRmf[ (454) in=40b,setxattr aYoTnltToaP\\jmjso_Y]Vqd[QmZjWZ[Y`eeQlQmf (1505) in=40b,setxattr nnjS`aUY\\iYXtal[`e]cX`P_nabr (454) in=28b,setxattr b\\n^fel`cdl`ZXRWVTT_enWnZrh]`PacpX[X[Raa (701) in=40b] snapc 2b=[2b,27,23,14] ondisk+write+known_if_redirected+supports_pool_eio e82)", , 2022-09-04T14:38:37.174 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:59.048885+0000", , 2022-09-04T14:38:37.174 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 217.73426515200001, Ignoring malparsed backtrace: 2022-09-04T14:38:37.197 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:915 3.fs0 3:fb230078:::smithi16339483-17:head [rollback 34] snapc 3b=[3b,3a,39,34,33] ondisk+write+known_if_redirected+supports_pool_eio e99)", , 2022-09-04T14:38:37.198 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:35:16.166595+0000", , 2022-09-04T14:38:37.198 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 200.61655500000001, Ignoring malparsed backtrace: 2022-09-04T14:38:37.221 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:916 3.fs0 3:fb230078:::smithi16339483-17:head [append 0~0] snapc 3b=[3b,3a,39,34,33] ondisk+write+known_if_redirected+supports_pool_eio e99)", , 2022-09-04T14:38:37.222 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:35:16.166637+0000", , 2022-09-04T14:38:37.222 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 200.616513201, Ignoring malparsed backtrace: 2022-09-04T14:38:37.245 INFO:teuthology.orchestra.run.smithi155.stdout:} , 2022-09-04T14:38:37.248 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight , 2022-09-04T14:38:37.245+0000 7f7962451700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 38719) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:38:37.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.721968+0000", , 2022-09-04T14:38:37.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7339999999999999e-06 , 2022-09-04T14:38:37.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:37.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.730989+0000", , 2022-09-04T14:38:37.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5683e-05 , 2022-09-04T14:38:37.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:37.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.759 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.759 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:62 3.ds0 3:b87fd6ef:::smithi16339483-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", Ignoring malparsed backtrace: 2022-09-04T14:38:37.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.784 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.785 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:73 3.ds0 3:b1abc525:::smithi16339483-25:head [append 0~409600,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", Ignoring malparsed backtrace: 2022-09-04T14:38:37.812 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:37.812 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 74 , 2022-09-04T14:38:37.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:37.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:37.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.761040+0000", , 2022-09-04T14:38:37.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00133603 Ignoring malparsed backtrace: 2022-09-04T14:38:37.858 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:37.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:37.858 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:37.882 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:38:37.883 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:37.883 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 110 Ignoring malparsed backtrace: 2022-09-04T14:38:37.905 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:38:37.905 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:37.906 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 111 Ignoring malparsed backtrace: 2022-09-04T14:38:37.927 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:38:37.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858781+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:37.949 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_started", , 2022-09-04T14:38:37.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.794945+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:37.972 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:37.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801246+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:37.993 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:37.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:38:37.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.543777+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:38.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.016 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:38.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:38:38.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:38:38.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.038 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:38.471 INFO:teuthology.orchestra.run.smithi155.stdout:{ , 2022-09-04T14:38:38.471 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, , 2022-09-04T14:38:38.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, Ignoring malparsed backtrace: 2022-09-04T14:38:38.501 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:38.501 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:38:38.502 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:38.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:38.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.538 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:38.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:38.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.569 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:38.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1722999999999999e-05 , 2022-09-04T14:38:38.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.593 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:38.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00095956199999999996 , 2022-09-04T14:38:38.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.618 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:38.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001069609 , 2022-09-04T14:38:38.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.641 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:38.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0180000000000004e-06 , 2022-09-04T14:38:38.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:38.664 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:38.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6855999999999999e-05 , 2022-09-04T14:38:38.693 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:38.693 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:38.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783673+0000", , 2022-09-04T14:38:38.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:38:38.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:38.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784665+0000", , 2022-09-04T14:38:38.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8630000000000008e-06 , 2022-09-04T14:38:38.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:38.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.794534+0000", , 2022-09-04T14:38:38.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093767590000000001 , 2022-09-04T14:38:38.770 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:38.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:38.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.830012+0000", , 2022-09-04T14:38:38.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:38.814 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:38.814 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.905044+0000", , 2022-09-04T14:38:38.814 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:38.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:38:38.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:56.032035+0000", , 2022-09-04T14:38:38.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:38:38.859 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:38.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.175821+0000", , 2022-09-04T14:38:38.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017596199999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:38.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:38:38.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:21.272124+0000", , 2022-09-04T14:38:38.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00323456 Ignoring malparsed backtrace: 2022-09-04T14:38:39.258 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops , 2022-09-04T14:38:39.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:38:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:38:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.760134+0000", , 2022-09-04T14:38:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0066417179999999996 Ignoring malparsed backtrace: 2022-09-04T14:38:39.452 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:38:39.452 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:39.452 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 157 Ignoring malparsed backtrace: 2022-09-04T14:38:39.484 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:39.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:39.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858431+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:39.511 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:39.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:39.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801919+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:39.535 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:39.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:39.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816182+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:39.560 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:38:39.561 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:39.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:38:39.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:39.595 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:39.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:38:39.625 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:39.625 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:38:39.625 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:39.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:38:39.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:39.649 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:39.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0049795409999999997 , 2022-09-04T14:38:39.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:39.672 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:39.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000875302 , 2022-09-04T14:38:39.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:39.695 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:39.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017441599999999999 , 2022-09-04T14:38:39.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:39.718 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:39.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.070060+0000", , 2022-09-04T14:38:39.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:39.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:39.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071722+0000", , 2022-09-04T14:38:39.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2679999999999994e-06 , 2022-09-04T14:38:39.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:39.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119564+0000", , 2022-09-04T14:38:39.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0695000000000001e-05 , 2022-09-04T14:38:39.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:39.815 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} , 2022-09-04T14:38:39.815 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} , 2022-09-04T14:38:39.815 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} Ignoring malparsed backtrace: 2022-09-04T14:38:41.729+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:38:41.763 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:38:41.764 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2022-09-04T14:38:42.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:38:42.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.818244+0000", , 2022-09-04T14:38:42.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:42.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:38:42.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.861348+0000", , 2022-09-04T14:38:42.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1210000000000001e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:42.130 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:42.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:38:42.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682473+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:42.160 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:42.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:38:42.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.741418+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:42.192 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:42.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:38:42.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910985+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:42.218 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:38:42.218 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", , 2022-09-04T14:38:42.219 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:38:42.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:42.244 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:42.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:42.268 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:42.268 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:38:42.268 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:42.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:38:42.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:42.293 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:42.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9359999999999994e-06 , 2022-09-04T14:38:42.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:42.318 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:42.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013434244 , 2022-09-04T14:38:42.342 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:42.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:42.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126141+0000", , 2022-09-04T14:38:42.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:38:42.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:42.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.131747+0000", , 2022-09-04T14:38:42.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2201000000000001e-05 , 2022-09-04T14:38:42.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:42.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.199165+0000", , 2022-09-04T14:38:42.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00022500999999999999 , 2022-09-04T14:38:42.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.199353+0000", , 2022-09-04T14:38:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6023e-05 , 2022-09-04T14:38:42.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:42.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:38:42.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.722763+0000", , 2022-09-04T14:38:42.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:42.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:38:42.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.724126+0000", , 2022-09-04T14:38:42.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.725e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:42.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:42.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.737326+0000", , 2022-09-04T14:38:42.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076465750000000001 Ignoring malparsed backtrace: 2022-09-04T14:38:43.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:43.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.738323+0000", , 2022-09-04T14:38:43.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083854519999999998 Ignoring malparsed backtrace: 2022-09-04T14:38:43.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:43.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.753392+0000", , 2022-09-04T14:38:43.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4231e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:43.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:38:43.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.762617+0000", , 2022-09-04T14:38:43.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00086305099999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:43.096 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:38:43.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.830556+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:43.128 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:38:43.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.860982+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:43.157 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:38:43.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.798464+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:43.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.181 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:43.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.209 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:38:43.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:38:43.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:38:43.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:38:43.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:38:43.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.257 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000147532 , 2022-09-04T14:38:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.281 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:43.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148901 , 2022-09-04T14:38:43.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.306 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:43.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:15.165460+0000", , 2022-09-04T14:38:43.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.454e-06 , 2022-09-04T14:38:43.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:38:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:38:43.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:38:43.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.874 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:43.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120665 , 2022-09-04T14:38:43.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.897 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6932e-05 , 2022-09-04T14:38:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.920 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:43.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910004+0000", , 2022-09-04T14:38:43.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:38:43.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:43.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:41.634557+0000", , 2022-09-04T14:38:43.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4173999999999998e-05 , 2022-09-04T14:38:43.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:43.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:43.987 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:43.987 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s1 74/33 ECSubWrite(tid=46, reqid=client.4557.0:638, at_version=74'31, trim_to=0'0, roll_forward_to=63'27))", Ignoring malparsed backtrace: 2022-09-04T14:38:44.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:38:44.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.087737+0000", , 2022-09-04T14:38:44.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018767009999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:44.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:44.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.090574+0000", , 2022-09-04T14:38:44.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4450000000000006e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:44.053 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:44.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:44.054 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:44.076 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:44.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:38:44.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.132521+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:44.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.133916+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:44.127 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:44.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:44.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:10.129917+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:38:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:38:44.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:44.526 INFO:teuthology.orchestra.run.smithi038.stdout:} , 2022-09-04T14:38:44.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops , 2022-09-04T14:38:44.572+0000 7ffa4ea67700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 38946) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:38:44.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:38:44.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:44.702 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:44.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024172000000000001 , 2022-09-04T14:38:44.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:44.727 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:44.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00067073199999999995 , 2022-09-04T14:38:44.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:44.751 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:44.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834327+0000", , 2022-09-04T14:38:44.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:38:44.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:44.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.777346+0000", , 2022-09-04T14:38:44.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0883e-05 , 2022-09-04T14:38:44.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:44.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.779695+0000", , 2022-09-04T14:38:44.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4693e-05 , 2022-09-04T14:38:44.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:44.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:38:44.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.908023+0000", , 2022-09-04T14:38:44.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:44.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:44.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910175+0000", , 2022-09-04T14:38:44.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4980000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:44.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:44.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910972+0000", , 2022-09-04T14:38:44.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6743000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:44.926 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:44.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:38:44.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954324+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:44.950 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:44.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:38:44.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954951+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:44.974 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:44.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:44.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.082269+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:45.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:45.010 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:45.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:45.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:45.041 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:45.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:45.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:45.072 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:45.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:46.467 INFO:teuthology.orchestra.run.smithi077.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:38:46.477 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight , 2022-09-04T14:38:46.492+0000 7f91016aa700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 39166) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:38:48.833 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:38:48.833 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:38:48.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:38:49.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.734112+0000", , 2022-09-04T14:38:49.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0869999999999998e-05 , 2022-09-04T14:38:49.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:49.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:49.158 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.158 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:69 3.9s0 3:97bfca98:::smithi16339483-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", Ignoring malparsed backtrace: 2022-09-04T14:38:49.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:38:49.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.774974+0000", , 2022-09-04T14:38:49.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:49.215 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:38:49.215 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.775016+0000", , 2022-09-04T14:38:49.215 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:49.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:38:49.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778525+0000", , 2022-09-04T14:38:49.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1900000000000003e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:49.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", , 2022-09-04T14:38:49.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.781033+0000", , 2022-09-04T14:38:49.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1576999999999995e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:49.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", , 2022-09-04T14:38:49.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.781151+0000", , 2022-09-04T14:38:49.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2256999999999994e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:49.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:49.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.791974+0000", , 2022-09-04T14:38:49.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010191699999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:49.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", , 2022-09-04T14:38:49.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.808940+0000", , 2022-09-04T14:38:49.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105621 Ignoring malparsed backtrace: 2022-09-04T14:38:49.357 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:38:49.358 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:49.358 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 208 Ignoring malparsed backtrace: 2022-09-04T14:38:49.381 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.381 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:38:49.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.787775+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:49.404 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:38:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:38:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:49.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:49.429 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:49.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:49.453 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:38:49.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:49.476 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:49.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:49.500 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:49.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:49.523 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:38:49.949 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776491+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:49.972 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:38:49.973 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776535+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:49.997 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:49.997 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:38:49.997 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833137+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:50.019 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:38:50.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843366+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:50.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.043 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:38:50.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.065 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:38:50.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.088 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:38:50.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:38:50.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.112 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:50.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2750000000000006e-05 , 2022-09-04T14:38:50.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.148 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:50.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2714000000000001e-05 , 2022-09-04T14:38:50.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.178 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:50.208 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.208 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs5 99/52 ECSubWrite(tid=117, reqid=client.4557.0:914, at_version=99'69, trim_to=0'0, roll_forward_to=89'67))", , 2022-09-04T14:38:50.208 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:16.170924+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:50.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.172902+0000", , 2022-09-04T14:38:50.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4032e-05 , 2022-09-04T14:38:50.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:50.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173459+0000", , 2022-09-04T14:38:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010605 , 2022-09-04T14:38:50.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:50.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.284 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.5s6 100/33 ECSubWrite(tid=70, reqid=client.4557.0:918, at_version=100'52, trim_to=0'0, roll_forward_to=97'50))", Ignoring malparsed backtrace: 2022-09-04T14:38:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:38:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:22.192248+0000", , 2022-09-04T14:38:50.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7877000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:38:50.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:38:45.016813+0000", , 2022-09-04T14:38:50.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2144999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.800802+0000", , 2022-09-04T14:38:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6996000000000001e-05 , 2022-09-04T14:38:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:38:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.803132+0000", , 2022-09-04T14:38:50.832 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:50.859 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:38:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.809562+0000", , 2022-09-04T14:38:50.860 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8646000000000005e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:50.882 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:50.882 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.780573+0000", , 2022-09-04T14:38:50.883 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4229999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:50.905 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.905 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:38:50.905 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683297+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:50.928 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.928 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:38:50.928 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.684065+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:50.951 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.951 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:38:50.951 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.697643+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:50.973 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.974 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.974 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:50.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:50.996 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:50.996 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:51.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:51.019 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:38:51.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:51.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:38:51.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:51.042 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:51.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0115000000000001e-05 , 2022-09-04T14:38:51.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:51.068 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:51.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017043200000000001 , 2022-09-04T14:38:51.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:38:51.090 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:51.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175461+0000", , 2022-09-04T14:38:51.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:38:51.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:51.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.271808+0000", , 2022-09-04T14:38:51.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4278e-05 , 2022-09-04T14:38:51.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:51.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:51.683 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:51.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:51.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:51.710 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:51.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:38:51.740 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:38:51.740 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:38:51.740 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:51.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.503e-06 , 2022-09-04T14:38:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:51.784 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:38:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:38:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:38:51.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910508+0000", , 2022-09-04T14:38:51.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7619000000000003e-05 , 2022-09-04T14:38:51.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:51.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.965089+0000", , 2022-09-04T14:38:51.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050946300000000004 , 2022-09-04T14:38:51.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:51.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:46.870324+0000", , 2022-09-04T14:38:51.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097045599999999997 , 2022-09-04T14:38:51.940 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:51.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:38:51.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.057569+0000", , 2022-09-04T14:38:51.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0399999999999993e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:38:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.121860+0000", , 2022-09-04T14:38:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000187896 Ignoring malparsed backtrace: 2022-09-04T14:38:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:38:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.077956+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:38:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126282+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:38:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.139041+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:38:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:38:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", Ignoring malparsed backtrace: osd.3: osd_enable_op_tracker = 'false' , osd.4: osd_enable_op_tracker = 'false' , osd.5: osd_enable_op_tracker = 'false' Ignoring malparsed backtrace: 2022-09-04T14:38:54.685 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} , 2022-09-04T14:38:54.695+0000 7f4052883700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 38941) UID: 0 , 2022-09-04T14:38:54.708 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} Ignoring malparsed backtrace: 2022-09-04T14:38:54.885 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 276.07465337999997, , 2022-09-04T14:38:54.885 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.009412264, , 2022-09-04T14:38:54.886 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:38:54.930 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:54.930 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:38:54.931 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.788278+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:54.961 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:54.961 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:38:54.962 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.815859+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:54.986 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:23.831072+0000", , 2022-09-04T14:38:54.986 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 271.02754989200002, , 2022-09-04T14:38:54.986 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0059717609999999999, Ignoring malparsed backtrace: 2022-09-04T14:38:55.011 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:55.011 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:55.011 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:38:55.036 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:55.037 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:55.037 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:38:55.070 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.6s6 47/41 ECSubWrite(tid=34, reqid=client.4557.0:304, at_version=0'0, trim_to=0'0, roll_forward_to=47'17))", , 2022-09-04T14:38:55.071 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:23.834171+0000", , 2022-09-04T14:38:55.071 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 271.02445072500001, Ignoring malparsed backtrace: 2022-09-04T14:38:55.106 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.9711000000000001e-05 , 2022-09-04T14:38:55.107 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:55.107 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:55.138 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.6744999999999999e-05 , 2022-09-04T14:38:55.138 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:55.139 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:55.164 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:55.164 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.es1 82/33 ECSubWrite(tid=21, reqid=client.4557.0:712, at_version=82'16, trim_to=0'0, roll_forward_to=58'14))", , 2022-09-04T14:38:55.165 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:59.063496+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:55.191 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.075897+0000", , 2022-09-04T14:38:55.191 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0014941080000000001 , 2022-09-04T14:38:55.191 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:55.216 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.082093+0000", , 2022-09-04T14:38:55.216 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00016168700000000001 , 2022-09-04T14:38:55.217 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:55.243 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:38:55.243 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:38:55.243 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.1s2 92/52 ECSubWrite(tid=258, reqid=client.4557.0:858, at_version=92'53, trim_to=0'0, roll_forward_to=90'51))", Ignoring malparsed backtrace: 2022-09-04T14:38:55.268 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:38:55.268 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192245+0000", , 2022-09-04T14:38:55.268 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.0753e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:55.292 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:55.292 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194506+0000", , 2022-09-04T14:38:55.292 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.7365000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:55.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.827353+0000", , 2022-09-04T14:38:55.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4549000000000007e-05 , 2022-09-04T14:38:55.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:55.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.829640+0000", , 2022-09-04T14:38:55.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.983e-05 , 2022-09-04T14:38:55.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:55.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.830351+0000", , 2022-09-04T14:38:55.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6718000000000001e-05 , 2022-09-04T14:38:55.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:55.823 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:55.823 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 288 , 2022-09-04T14:38:55.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:55.847 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:38:55.848 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 289 , 2022-09-04T14:38:55.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:38:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.830330+0000", , 2022-09-04T14:38:55.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012380299999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:55.895 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:55.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:55.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.772627+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:55.919 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:55.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:55.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.777066+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:55.948 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:55.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:55.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.777102+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:55.978 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:55.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:55.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.777131+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:56.004 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:38:56.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:27.797058+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:56.028 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", , 2022-09-04T14:38:56.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.067615+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:56.052 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", , 2022-09-04T14:38:56.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.067677+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:56.075 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:38:56.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.069669+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:56.098 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:38:56.098 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:38:56.098 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:38:56.124 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:38:56.124 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:38:56.124 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:38:56.147 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:38:56.148 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:38:56.148 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:38:56.177 INFO:teuthology.orchestra.run.smithi038.stdout: , dumped all , 2022-09-04T14:38:56.179 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2022-09-04T14:38:55.720209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135987744,"num_objects":179,"num_object_clones":127,"num_object_copies":1243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":69,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":14,"num_read":569,"num_read_kb":218307,"num_write":1077,"num_write_kb":142660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":936,"ondisk_log_size":936,"up":127,"acting":127,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":1499190080,"kb_used":13218240,"kb_used_data":1485972044,"kb_used_omap":32241,"kb_used_meta":0,"kb_avail":1485971840,"statfs":{"total":1535170641920,"available":1521635164160,"internally_reserved":208896,"allocated":1521635373056,"data_stored":1521635373056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33015213,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.343262"},"pg_stats":[{"pgid":"3.f","version":"102'73","reported_seq":323,"reported_epoch":203,"state":"active+undersized+degraded","last_fresh":"2022-09-04T14:38:49.443750+0000","last_change":"2022-09-04T14:34:29.678697+0000","last_active":"2022-09-04T14:38:49.443750+0000","last_peered":"2022-09-04T14:38:49.443750+0000","last_clean":"2022-09-04T14:34:15.747859+0000","last_became_active":"2022-09-04T14:34:29.678697+0000","last_became_peered":"2022-09-04T14:34:29.678697+0000","last_unstale":"2022-09-04T14:38:49.443750+0000","last_undegraded":"2022-09-04T14:34:29.670244+0000","last_fullsized":"2022-09-04T14:34:29.669996+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:34:09.681013+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014105599999999999,"stat_sum":{"num_bytes":15302656,"num_objects":21,"num_object_clones":15,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":50,"num_read_kb":27741,"num_write":151,"num_write_kb":14944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,15,2147483647,8,11,0],"acting":[2,3,15,2147483647,8,11,0],"avail_no_missing":["2(0)","0(6)","3(1)","8(4)","11(5)","15(2)"],"object_location_counts":[{"shards":"0(6),2(0),3(1),8(4),11(5),15(2)","objects":21}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"106'27","reported_seq":849,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.359424+0000","last_change":"2022-09-04T14:38:53.359595+0000","last_active":"2022-09-04T14:38:53.359424+0000","last_peered":"2022-09-04T14:38:53.359424+0000","last_clean":"2022-09-04T14:38:53.359424+0000","last_became_active":"2022-09-04T14:34:11.056319+0000","last_became_peered":"2022-09-04T14:34:11.056319+0000","last_unstale":"2022-09-04T14:38:53.359424+0000","last_undegraded":"2022-09-04T14:38:53.359424+0000","last_fullsized":"2022-09-04T14:38:53.359424+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'27","last_scrub_stamp":"2022-09-04T14:37:30.412644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:37:30.412644+0000","objects_scrubbed":4,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":28,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011181953999999999,"objects_trimmed":0,"snaptrim_duration":0.000353464,"stat_sum":{"num_bytes":3620864,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":4564,"num_write":14,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1,10,8,3,12],"acting":[11,9,1,10,8,3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.d","version":"97'57","reported_seq":898,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.359169+0000","last_change":"2022-09-04T14:38:53.359277+0000","last_active":"2022-09-04T14:38:53.359169+0000","last_peered":"2022-09-04T14:38:53.359169+0000","last_clean":"2022-09-04T14:38:53.359169+0000","last_became_active":"2022-09-04T14:34:11.155525+0000","last_became_peered":"2022-09-04T14:34:11.155525+0000","last_unstale":"2022-09-04T14:38:53.359169+0000","last_undegraded":"2022-09-04T14:38:53.359169+0000","last_fullsized":"2022-09-04T14:38:53.359169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'57","last_scrub_stamp":"2022-09-04T14:38:28.599911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:38:28.599911+0000","objects_scrubbed":11,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":28,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:51.132232+0000","scrub_duration":0.0057081010000000001,"objects_trimmed":0,"snaptrim_duration":0.00042639700000000002,"stat_sum":{"num_bytes":10305536,"num_objects":11,"num_object_clones":7,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":24,"num_read_kb":13451,"num_write":57,"num_write_kb":11168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,11,15,7,8,4],"acting":[6,2,11,15,7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.c","version":"107'41","reported_seq":862,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:52.722595+0000","last_change":"2022-09-04T14:38:52.722681+0000","last_active":"2022-09-04T14:38:52.722595+0000","last_peered":"2022-09-04T14:38:52.722595+0000","last_clean":"2022-09-04T14:38:52.722595+0000","last_became_active":"2022-09-04T14:34:10.867118+0000","last_became_peered":"2022-09-04T14:34:10.867118+0000","last_unstale":"2022-09-04T14:38:52.722595+0000","last_undegraded":"2022-09-04T14:38:52.722595+0000","last_fullsized":"2022-09-04T14:38:52.722595+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'41","last_scrub_stamp":"2022-09-04T14:38:42.568004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:38:42.568004+0000","objects_scrubbed":7,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":28,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:58.890505+0000","scrub_duration":0.0069564079999999999,"objects_trimmed":0,"snaptrim_duration":0.0066303910000000002,"stat_sum":{"num_bytes":4915200,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":17978,"num_write":26,"num_write_kb":6640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6,3,7,12,5],"acting":[14,8,6,3,7,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"2"},{"start":"b","length":"1"}]},{"pgid":"3.b","version":"105'52","reported_seq":905,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.359410+0000","last_change":"2022-09-04T14:38:53.359497+0000","last_active":"2022-09-04T14:38:53.359410+0000","last_peered":"2022-09-04T14:38:53.359410+0000","last_clean":"2022-09-04T14:38:53.359410+0000","last_became_active":"2022-09-04T14:34:11.055874+0000","last_became_peered":"2022-09-04T14:34:11.055874+0000","last_unstale":"2022-09-04T14:38:53.359410+0000","last_undegraded":"2022-09-04T14:38:53.359410+0000","last_fullsized":"2022-09-04T14:38:53.359410+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'52","last_scrub_stamp":"2022-09-04T14:38:45.024329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:38:45.024329+0000","objects_scrubbed":11,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:40:07.292985+0000","scrub_duration":0.34571597199999998,"objects_trimmed":0,"snaptrim_duration":0.0049398189999999998,"stat_sum":{"num_bytes":10797056,"num_objects":11,"num_object_clones":8,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":41,"num_read_kb":32185,"num_write":53,"num_write_kb":10544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,11,10,9],"acting":[4,0,1,12,11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"2"}]},{"pgid":"3.a","version":"81'58","reported_seq":923,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.359543+0000","last_change":"2022-09-04T14:38:53.359625+0000","last_active":"2022-09-04T14:38:53.359543+0000","last_peered":"2022-09-04T14:38:53.359543+0000","last_clean":"2022-09-04T14:38:53.359543+0000","last_became_active":"2022-09-04T14:34:11.156143+0000","last_became_peered":"2022-09-04T14:34:11.156143+0000","last_unstale":"2022-09-04T14:38:53.359543+0000","last_undegraded":"2022-09-04T14:38:53.359543+0000","last_fullsized":"2022-09-04T14:38:53.359543+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'58","last_scrub_stamp":"2022-09-04T14:38:07.591860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:38:07.591860+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":32,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:15.522867+0000","scrub_duration":0.0053977779999999998,"objects_trimmed":0,"snaptrim_duration":0.0011658790000000001,"stat_sum":{"num_bytes":10862592,"num_objects":11,"num_object_clones":7,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":26,"num_read_kb":685,"num_write":40,"num_write_kb":10752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,1,14,3,15,2],"acting":[6,10,1,14,3,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"3.9","version":"106'100","reported_seq":945,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.359315+0000","last_change":"2022-09-04T14:38:53.359441+0000","last_active":"2022-09-04T14:38:53.359315+0000","last_peered":"2022-09-04T14:38:53.359315+0000","last_clean":"2022-09-04T14:38:53.359315+0000","last_became_active":"2022-09-04T14:34:11.056214+0000","last_became_peered":"2022-09-04T14:34:11.056214+0000","last_unstale":"2022-09-04T14:38:53.359315+0000","last_undegraded":"2022-09-04T14:38:53.359315+0000","last_fullsized":"2022-09-04T14:38:53.359315+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'100","last_scrub_stamp":"2022-09-04T14:38:18.770490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:38:18.770490+0000","objects_scrubbed":23,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:33.266371+0000","scrub_duration":0.014715018999999999,"objects_trimmed":0,"snaptrim_duration":0.018020023,"stat_sum":{"num_bytes":13287424,"num_objects":23,"num_object_clones":17,"num_object_copies":161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":64,"num_read_kb":33341,"num_write":120,"num_write_kb":14448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0,11,14,8,3],"acting":[7,1,0,11,14,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"2"}]},{"pgid":"3.8","version":"0'0","reported_seq":360,"reported_epoch":203,"state":"active+clean","last_fresh":"2022-09-04T14:38:49.957923+0000","last_change":"2022-09-04T14:38:04.688049+0000","last_active":"2022-09-04T14:38:49.957923+0000","last_peered":"2022-09-04T14:38:49.957923+0000","last_clean":"2022-09-04T14:38:49.957923+0000","last_became_active":"2022-09-04T14:34:10.697448+0000","last_became_peered":"2022-09-04T14:34:10.697448+0000","last_unstale":"2022-09-04T14:38:49.957923+0000","last_undegraded":"2022-09-04T14:38:49.957923+0000","last_fullsized":"2022-09-04T14:38:49.957923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:38:04.687976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:38:04.687976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:24.798426+0000","scrub_duration":0.0022344550000000002,"objects_trimmed":0,"snaptrim_duration":0.00030780500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,11,3,14,2,7],"acting":[5,0,11,3,14,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"3"},{"start":"b","length":"2"},{"start":"e","length":"1"},{"start":"11","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"},{"start":"1e","length":"2"},{"start":"22","length":"2"},{"start":"25","length":"1"},{"start":"27","length":"1"},{"start":"2b","length":"1"},{"start":"2d","length":"3"},{"start":"33","length":"2"},{"start":"36","length":"1"},{"start":"39","length":"3"},{"start":"42","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":222,"reported_epoch":203,"state":"active+clean","last_fresh":"2022-09-04T14:38:50.579581+0000","last_change":"2022-09-04T14:37:43.930720+0000","last_active":"2022-09-04T14:38:50.579581+0000","last_peered":"2022-09-04T14:38:50.579581+0000","last_clean":"2022-09-04T14:38:50.579581+0000","last_became_active":"2022-09-04T14:33:58.330284+0000","last_became_peered":"2022-09-04T14:33:58.330284+0000","last_unstale":"2022-09-04T14:38:50.579581+0000","last_undegraded":"2022-09-04T14:38:50.579581+0000","last_fullsized":"2022-09-04T14:38:50.579581+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:37:43.930656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_clean_scrub_stamp":"2022-09-04T14:37:43.930656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:06.790763+0000","scrub_duration":0.0016927730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"107'54","reported_seq":897,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.289899+0000","last_change":"2022-09-04T14:38:53.289995+0000","last_active":"2022-09-04T14:38:53.289899+0000","last_peered":"2022-09-04T14:38:53.289899+0000","last_clean":"2022-09-04T14:38:53.289899+0000","last_became_active":"2022-09-04T14:34:10.866976+0000","last_became_peered":"2022-09-04T14:34:10.866976+0000","last_unstale":"2022-09-04T14:38:53.289899+0000","last_undegraded":"2022-09-04T14:38:53.289899+0000","last_fullsized":"2022-09-04T14:38:53.289899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'54","last_scrub_stamp":"2022-09-04T14:38:06.879610+0000","last_deep_scrub":"107'54","last_deep_scrub_stamp":"2022-09-04T14:36:34.233077+0000","last_clean_scrub_stamp":"2022-09-04T14:38:06.879610+0000","objects_scrubbed":12,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:11.547778+0000","scrub_duration":0.014607258,"objects_trimmed":0,"snaptrim_duration":0.0081453529999999993,"stat_sum":{"num_bytes":8749056,"num_objects":12,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":25,"num_read_kb":7208,"num_write":31,"num_write_kb":8592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4,14,12,0,11],"acting":[10,8,4,14,12,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":224,"reported_epoch":203,"state":"active+clean","last_fresh":"2022-09-04T14:38:49.443593+0000","last_change":"2022-09-04T14:38:05.184228+0000","last_active":"2022-09-04T14:38:49.443593+0000","last_peered":"2022-09-04T14:38:49.443593+0000","last_clean":"2022-09-04T14:38:49.443593+0000","last_became_active":"2022-09-04T14:33:58.159320+0000","last_became_peered":"2022-09-04T14:33:58.159320+0000","last_unstale":"2022-09-04T14:38:49.443593+0000","last_undegraded":"2022-09-04T14:38:49.443593+0000","last_fullsized":"2022-09-04T14:38:49.443593+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:38:05.184160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_clean_scrub_stamp":"2022-09-04T14:38:05.184160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:08.440134+0000","scrub_duration":0.0024755889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":193,"reported_epoch":203,"state":"active+undersized","last_fresh":"2022-09-04T14:38:49.443597+0000","last_change":"2022-09-04T14:34:29.675043+0000","last_active":"2022-09-04T14:38:49.443597+0000","last_peered":"2022-09-04T14:38:49.443597+0000","last_clean":"2022-09-04T14:34:09.681013+0000","last_became_active":"2022-09-04T14:34:29.675043+0000","last_became_peered":"2022-09-04T14:34:29.675043+0000","last_unstale":"2022-09-04T14:38:49.443597+0000","last_undegraded":"2022-09-04T14:38:49.443597+0000","last_fullsized":"2022-09-04T14:34:29.669913+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:34:09.681013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,4,14,11,5,10],"acting":[2147483647,2,4,14,11,5,10],"avail_no_missing":["2(1)","4(2)","5(5)","10(6)","11(4)","14(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":173,"reported_epoch":203,"state":"active+undersized","last_fresh":"2022-09-04T14:38:49.958026+0000","last_change":"2022-09-04T14:34:29.669232+0000","last_active":"2022-09-04T14:38:49.958026+0000","last_peered":"2022-09-04T14:38:49.958026+0000","last_clean":"2022-09-04T14:33:57.033083+0000","last_became_active":"2022-09-04T14:34:29.669232+0000","last_became_peered":"2022-09-04T14:34:29.669232+0000","last_unstale":"2022-09-04T14:38:49.958026+0000","last_undegraded":"2022-09-04T14:38:49.958026+0000","last_fullsized":"2022-09-04T14:34:29.668873+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_clean_scrub_stamp":"2022-09-04T14:33:57.033083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"107'74","reported_seq":927,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.358971+0000","last_change":"2022-09-04T14:38:53.359299+0000","last_active":"2022-09-04T14:38:53.358971+0000","last_peered":"2022-09-04T14:38:53.358971+0000","last_clean":"2022-09-04T14:38:53.358971+0000","last_became_active":"2022-09-04T14:34:11.155200+0000","last_became_peered":"2022-09-04T14:34:11.155200+0000","last_unstale":"2022-09-04T14:38:53.358971+0000","last_undegraded":"2022-09-04T14:38:53.358971+0000","last_fullsized":"2022-09-04T14:38:53.358971+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'74","last_scrub_stamp":"2022-09-04T14:37:58.301856+0000","last_deep_scrub":"107'74","last_deep_scrub_stamp":"2022-09-04T14:37:58.301856+0000","last_clean_scrub_stamp":"2022-09-04T14:37:58.301856+0000","objects_scrubbed":12,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:10.751348+0000","scrub_duration":0.073120270000000001,"objects_trimmed":0,"snaptrim_duration":0.010208480000000001,"stat_sum":{"num_bytes":14417920,"num_objects":12,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":36,"num_read_kb":21088,"num_write":46,"num_write_kb":16352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,1,5,15,4],"acting":[3,7,6,1,5,15,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":222,"reported_epoch":203,"state":"active+clean","last_fresh":"2022-09-04T14:38:49.957905+0000","last_change":"2022-09-04T14:38:09.691587+0000","last_active":"2022-09-04T14:38:49.957905+0000","last_peered":"2022-09-04T14:38:49.957905+0000","last_clean":"2022-09-04T14:38:49.957905+0000","last_became_active":"2022-09-04T14:33:58.046366+0000","last_became_peered":"2022-09-04T14:33:58.046366+0000","last_unstale":"2022-09-04T14:38:49.957905+0000","last_undegraded":"2022-09-04T14:38:49.957905+0000","last_fullsized":"2022-09-04T14:38:49.957905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:38:09.691534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:36:43.610906+0000","last_clean_scrub_stamp":"2022-09-04T14:38:09.691534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:10.196755+0000","scrub_duration":0.0017539890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'157","reported_seq":375,"reported_epoch":203,"state":"active+undersized+degraded","last_fresh":"2022-09-04T14:38:49.446230+0000","last_change":"2022-09-04T14:34:29.672296+0000","last_active":"2022-09-04T14:38:49.446230+0000","last_peered":"2022-09-04T14:38:49.446230+0000","last_clean":"2022-09-04T14:33:53.852377+0000","last_became_active":"2022-09-04T14:34:29.672296+0000","last_became_peered":"2022-09-04T14:34:29.672296+0000","last_unstale":"2022-09-04T14:38:49.446230+0000","last_undegraded":"2022-09-04T14:34:29.671914+0000","last_fullsized":"2022-09-04T14:34:29.671820+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:33:50.991721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:50.991721+0000","last_clean_scrub_stamp":"2022-09-04T14:33:50.991721+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15],"acting":[15],"avail_no_missing":["15"],"object_location_counts":[{"shards":"15","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.2","version":"104'31","reported_seq":258,"reported_epoch":203,"state":"active+undersized+degraded","last_fresh":"2022-09-04T14:38:49.444298+0000","last_change":"2022-09-04T14:34:29.674277+0000","last_active":"2022-09-04T14:38:49.444298+0000","last_peered":"2022-09-04T14:38:49.444298+0000","last_clean":"2022-09-04T14:34:13.145309+0000","last_became_active":"2022-09-04T14:34:29.674277+0000","last_became_peered":"2022-09-04T14:34:29.674277+0000","last_unstale":"2022-09-04T14:38:49.444298+0000","last_undegraded":"2022-09-04T14:34:29.671016+0000","last_fullsized":"2022-09-04T14:34:29.670551+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:34:09.681013+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7127040,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":17,"num_read_kb":12198,"num_write":22,"num_write_kb":7456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,6,10,14,1,7],"acting":[2147483647,3,6,10,14,1,7],"avail_no_missing":["3(1)","1(5)","6(2)","7(6)","10(3)","14(4)"],"object_location_counts":[{"shards":"1(5),3(1),6(2),7(6),10(3),14(4)","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":224,"reported_epoch":203,"state":"active+clean","last_fresh":"2022-09-04T14:38:49.443519+0000","last_change":"2022-09-04T14:37:53.191184+0000","last_active":"2022-09-04T14:38:49.443519+0000","last_peered":"2022-09-04T14:38:49.443519+0000","last_clean":"2022-09-04T14:38:49.443519+0000","last_became_active":"2022-09-04T14:33:58.236501+0000","last_became_peered":"2022-09-04T14:33:58.236501+0000","last_unstale":"2022-09-04T14:38:49.443519+0000","last_undegraded":"2022-09-04T14:38:49.443519+0000","last_fullsized":"2022-09-04T14:38:49.443519+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:37:53.191076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_clean_scrub_stamp":"2022-09-04T14:37:53.191076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:07.332876+0000","scrub_duration":0.001676505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"107'59","reported_seq":294,"reported_epoch":203,"state":"active+undersized+degraded","last_fresh":"2022-09-04T14:38:50.579502+0000","last_change":"2022-09-04T14:34:29.676348+0000","last_active":"2022-09-04T14:38:50.579502+0000","last_peered":"2022-09-04T14:38:50.579502+0000","last_clean":"2022-09-04T14:34:16.240161+0000","last_became_active":"2022-09-04T14:34:29.676348+0000","last_became_peered":"2022-09-04T14:34:29.676348+0000","last_unstale":"2022-09-04T14:38:50.579502+0000","last_undegraded":"2022-09-04T14:34:29.672513+0000","last_fullsized":"2022-09-04T14:34:29.672189+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:34:09.681013+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000468837,"stat_sum":{"num_bytes":11190272,"num_objects":16,"num_object_clones":13,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":25,"num_read_kb":19866,"num_write":81,"num_write_kb":10928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14,9,5,6,3,2147483647],"acting":[4,14,9,5,6,3,2147483647],"avail_no_missing":["4(0)","3(5)","5(3)","6(4)","9(2)","14(1)"],"object_location_counts":[{"shards":"3(5),4(0),5(3),6(4),9(2),14(1)","objects":16}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":219,"reported_epoch":203,"state":"active+clean","last_fresh":"2022-09-04T14:38:50.776881+0000","last_change":"2022-09-04T14:37:42.962852+0000","last_active":"2022-09-04T14:38:50.776881+0000","last_peered":"2022-09-04T14:38:50.776881+0000","last_clean":"2022-09-04T14:38:50.776881+0000","last_became_active":"2022-09-04T14:33:58.053151+0000","last_became_peered":"2022-09-04T14:33:58.053151+0000","last_unstale":"2022-09-04T14:38:50.776881+0000","last_undegraded":"2022-09-04T14:38:50.776881+0000","last_fullsized":"2022-09-04T14:38:50.776881+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:37:42.962783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_clean_scrub_stamp":"2022-09-04T14:37:42.962783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:38:57.311510+0000","scrub_duration":0.0018951059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"107'36","reported_seq":287,"reported_epoch":203,"state":"active+undersized+degraded","last_fresh":"2022-09-04T14:38:50.671429+0000","last_change":"2022-09-04T14:34:29.678431+0000","last_active":"2022-09-04T14:38:50.671429+0000","last_peered":"2022-09-04T14:38:50.671429+0000","last_clean":"2022-09-04T14:34:27.662667+0000","last_became_active":"2022-09-04T14:34:29.678431+0000","last_became_peered":"2022-09-04T14:34:29.678431+0000","last_unstale":"2022-09-04T14:38:50.671429+0000","last_undegraded":"2022-09-04T14:34:29.672733+0000","last_fullsized":"2022-09-04T14:34:29.672517+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:34:09.681013+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014158700000000001,"stat_sum":{"num_bytes":7995392,"num_objects":9,"num_object_clones":7,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":24,"num_read_kb":5290,"num_write":66,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,15,4,1,0,3,2147483647],"acting":[8,15,4,1,0,3,2147483647],"avail_no_missing":["8(0)","0(4)","1(3)","3(5)","4(2)","15(1)"],"object_location_counts":[{"shards":"0(4),1(3),3(5),4(2),8(0),15(1)","objects":9}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"2"}]},{"pgid":"3.7","version":"97'76","reported_seq":920,"reported_epoch":204,"state":"active+clean+snaptrim_error","last_fresh":"2022-09-04T14:38:53.359614+0000","last_change":"2022-09-04T14:38:53.359689+0000","last_active":"2022-09-04T14:38:53.359614+0000","last_peered":"2022-09-04T14:38:53.359614+0000","last_clean":"2022-09-04T14:38:53.359614+0000","last_became_active":"2022-09-04T14:34:10.933166+0000","last_became_peered":"2022-09-04T14:34:10.933166+0000","last_unstale":"2022-09-04T14:38:53.359614+0000","last_undegraded":"2022-09-04T14:38:53.359614+0000","last_fullsized":"2022-09-04T14:38:53.359614+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'76","last_scrub_stamp":"2022-09-04T14:38:05.955784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:38:05.955784+0000","objects_scrubbed":19,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:20.015617+0000","scrub_duration":0.0089357580000000002,"objects_trimmed":0,"snaptrim_duration":0.77139303100000001,"stat_sum":{"num_bytes":8896512,"num_objects":19,"num_object_clones":14,"num_object_copies":133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":37,"num_read_kb":16989,"num_write":99,"num_write_kb":8144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,0,8,5,14],"acting":[4,11,9,0,8,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":203,"state":"active+clean","last_fresh":"2022-09-04T14:38:50.671578+0000","last_change":"2022-09-04T14:38:18.167129+0000","last_active":"2022-09-04T14:38:50.671578+0000","last_peered":"2022-09-04T14:38:50.671578+0000","last_clean":"2022-09-04T14:38:50.671578+0000","last_became_active":"2022-09-04T14:33:58.267488+0000","last_became_peered":"2022-09-04T14:33:58.267488+0000","last_unstale":"2022-09-04T14:38:50.671578+0000","last_undegraded":"2022-09-04T14:38:50.671578+0000","last_fullsized":"2022-09-04T14:38:50.671578+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:38:18.167059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_clean_scrub_stamp":"2022-09-04T14:38:18.167059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-09-04T14:39:48.075246+0000","scrub_duration":0.0021853689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":205,"reported_epoch":203,"state":"active+undersized","last_fresh":"2022-09-04T14:38:50.907636+0000","last_change":"2022-09-04T14:34:29.672394+0000","last_active":"2022-09-04T14:38:50.907636+0000","last_peered":"2022-09-04T14:38:50.907636+0000","last_clean":"2022-09-04T14:34:27.660218+0000","last_became_active":"2022-09-04T14:34:29.672394+0000","last_became_peered":"2022-09-04T14:34:29.672394+0000","last_unstale":"2022-09-04T14:38:50.907636+0000","last_undegraded":"2022-09-04T14:38:50.907636+0000","last_fullsized":"2022-09-04T14:34:29.672040+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:33:57.033083+0000","last_clean_scrub_stamp":"2022-09-04T14:33:57.033083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"104'41","reported_seq":284,"reported_epoch":203,"state":"active+undersized+degraded","last_fresh":"2022-09-04T14:38:50.576206+0000","last_change":"2022-09-04T14:34:29.676245+0000","last_active":"2022-09-04T14:38:50.576206+0000","last_peered":"2022-09-04T14:38:50.576206+0000","last_clean":"2022-09-04T14:34:15.752047+0000","last_became_active":"2022-09-04T14:34:29.676245+0000","last_became_peered":"2022-09-04T14:34:29.676245+0000","last_unstale":"2022-09-04T14:38:50.576206+0000","last_undegraded":"2022-09-04T14:34:29.673347+0000","last_fullsized":"2022-09-04T14:34:29.673141+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-04T14:34:09.681013+0000","last_clean_scrub_stamp":"2022-09-04T14:34:09.681013+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032131800000000002,"stat_sum":{"num_bytes":7929856,"num_objects":12,"num_object_clones":7,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":34,"num_read_kb":5605,"num_write":68,"num_write_kb":7744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2147483647,1,3,6,2,9],"acting":[0,2147483647,1,3,6,2,9],"avail_no_missing":["0(0)","1(2)","2(5)","3(3)","6(4)","9(6)"],"object_location_counts":[{"shards":"0(0),1(2),2(5),3(3),6(4),9(6)","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":135397376,"num_objects":177,"num_object_clones":127,"num_object_copies":1239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":67,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":14,"num_read":433,"num_read_kb":218189,"num_write":874,"num_write_kb":139200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":779,"ondisk_log_size":779,"up":112,"acting":112,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":0}],"osd_stats":[{"osd":12,"up_from":26,"seq":111669149759,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":813496,"kb_used_data":92885896,"kb_used_omap":1310,"kb_used_meta":0,"kb_avail":92885884,"statfs":{"total":95948165120,"available":95115145216,"internally_reserved":12288,"allocated":95115157504,"data_stored":95115157504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1341676,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.65700000000000003,"15min":0.92500000000000004},"min":{"1min":0.26500000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.72399999999999998,"5min":12.435,"15min":12.435},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.59599999999999997,"15min":0.75600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.85699999999999998,"5min":7.2030000000000003,"15min":7.2030000000000003},"last":0.441}]},{"osd":1,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":7.7460000000000004,"15min":14.954000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81399999999999995,"5min":299.91000000000003,"15min":299.91000000000003},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59199999999999997,"15min":0.69199999999999995},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76200000000000001,"5min":4.2930000000000001,"15min":4.2930000000000001},"last":0.314}]},{"osd":2,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":2.7730000000000001,"15min":1.5549999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94899999999999995,"5min":204.56999999999999,"15min":204.56999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.59399999999999997,"15min":0.68400000000000005},"min":{"1min":0.32200000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.75600000000000001,"5min":4.2699999999999996,"15min":4.2699999999999996},"last":0.38}]},{"osd":3,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":20.221,"5min":6.6529999999999996,"15min":2.9060000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":414.55500000000001,"5min":414.55500000000001,"15min":414.55500000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":4.2969999999999997,"15min":2.0990000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85199999999999998,"5min":228.53800000000001,"15min":228.53800000000001},"last":0.56499999999999995}]},{"osd":4,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":10.928000000000001,"15min":4.2199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71899999999999997,"5min":500.84199999999998,"15min":500.84199999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":35.369999999999997,"5min":14.337999999999999,"15min":5.4219999999999997},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":732.37300000000005,"5min":732.37300000000005,"15min":732.37300000000005},"last":0.41999999999999998}]},{"osd":5,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.63100000000000001,"15min":0.72999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88800000000000001,"5min":4.3090000000000002,"15min":4.3090000000000002},"last":0.372},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.72999999999999998,"15min":0.92200000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0389999999999999,"5min":8.2970000000000006,"15min":8.2970000000000006},"last":0.35999999999999999}]},{"osd":6,"last update":"Sun Sep 4 14:38:43 2022","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":14.164999999999999,"15min":27.047999999999998},"min":{"1min":0.46300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.85499999999999998,"5min":560.08199999999999,"15min":560.08199999999999},"last":0.432},{"interface":"front","average":{"1min":20.841999999999999,"5min":4.633,"15min":1.9359999999999999},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":485.48399999999998,"5min":485.48399999999998,"15min":485.48399999999998},"last":0.41399999999999998}]},{"osd":11,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.78600000000000003,"15min":0.89500000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0880000000000001,"5min":4.3220000000000001,"15min":4.3220000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":3.722,"15min":6.8259999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.97299999999999998,"5min":124.03400000000001,"15min":124.03400000000001},"last":0.64800000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":14,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.874,"15min":1.0760000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.109,"5min":8.2469999999999999,"15min":8.2469999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":22.657,"15min":33.222999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.119,"5min":684.47000000000003,"15min":684.47000000000003},"last":0.58699999999999997}]},{"osd":15,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":24.474,"5min":5.5270000000000001,"15min":2.4670000000000001},"min":{"1min":0.45600000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":501.61700000000002,"5min":501.61700000000002,"15min":501.61700000000002},"last":0.627},{"interface":"front","average":{"1min":30.576000000000001,"5min":14.356,"15min":5.5369999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":501.65600000000001,"5min":715.63499999999999,"15min":715.63499999999999},"last":0.54500000000000004}]}]},{"osd":15,"up_from":21,"seq":90194313281,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":828612,"kb_used_data":92870780,"kb_used_omap":1588,"kb_used_meta":0,"kb_avail":92870768,"statfs":{"total":95948165120,"available":95099666432,"internally_reserved":12288,"allocated":95099678720,"data_stored":95099678720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1626464,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":30.428000000000001,"5min":6.5350000000000001,"15min":2.5369999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":570.02300000000002,"5min":570.02300000000002,"15min":570.02300000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":49.619999999999997,"5min":13.292,"15min":4.7549999999999999},"min":{"1min":0.32100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":934.82000000000005,"5min":934.82000000000005,"15min":934.82000000000005},"last":229.66399999999999}]},{"osd":1,"last update":"Sun Sep 4 14:38:37 2022","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.65900000000000003,"15min":0.69099999999999995},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81999999999999995,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57399999999999995}]},{"osd":2,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54700000000000004,"15min":0.55700000000000005},"min":{"1min":0.38900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80900000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":6.5490000000000004,"15min":3.9569999999999999},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":387.55799999999999,"15min":387.55799999999999},"last":0.627}]},{"osd":3,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":26.620999999999999,"5min":14.266999999999999,"15min":19.306999999999999},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":445.62299999999999,"5min":445.62299999999999,"15min":445.62299999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.438,"5min":4.8140000000000001,"15min":9.1799999999999997},"min":{"1min":0.308,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.625,"5min":207.50200000000001,"15min":207.50200000000001},"last":0.73699999999999999}]},{"osd":4,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80900000000000005,"5min":3.1520000000000001,"15min":3.1520000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":19.786999999999999,"5min":4.4459999999999997,"15min":1.903},"min":{"1min":0.36899999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":347.17399999999998,"5min":347.17399999999998,"15min":347.17399999999998},"last":0.755}]},{"osd":5,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.64300000000000002,"15min":0.69899999999999995},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.88900000000000001,"5min":3.8940000000000001,"15min":3.8940000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":5.5199999999999996,"15min":2.3719999999999999},"min":{"1min":0.44,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.85299999999999998,"5min":457.476,"15min":457.476},"last":0.68999999999999995}]},{"osd":6,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":2.0339999999999998,"15min":1.226},"min":{"1min":0.44500000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76500000000000001,"5min":120.952,"15min":120.952},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":14.782,"15min":5.3280000000000003},"min":{"1min":0.443,"5min":0.375,"15min":0.375},"max":{"1min":0.81499999999999995,"5min":923.70299999999997,"15min":923.70299999999997},"last":0.79200000000000004}]},{"osd":7,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86199999999999999,"5min":3.069,"15min":3.069},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.36899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.86399999999999999}]},{"osd":8,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":23.027999999999999,"5min":13.929,"15min":5.0670000000000002},"min":{"1min":0.441,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":403.36500000000001,"5min":746.78200000000004,"15min":746.78200000000004},"last":0.873},{"interface":"front","average":{"1min":47.222999999999999,"5min":9.9849999999999994,"15min":3.8039999999999998},"min":{"1min":0.44,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":839.36300000000006,"5min":839.36300000000006,"15min":839.36300000000006},"last":0.88900000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":14,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":7.5469999999999997,"15min":3.351},"min":{"1min":0.52600000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89200000000000002,"5min":599.88499999999999,"15min":599.88499999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":15.464,"15min":5.7610000000000001},"min":{"1min":0.51200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94999999999999996,"5min":920.78700000000003,"15min":920.78700000000003},"last":0.88}]}]},{"osd":14,"up_from":21,"seq":90194313280,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":829492,"kb_used_data":92869900,"kb_used_omap":2695,"kb_used_meta":0,"kb_avail":92869888,"statfs":{"total":95948165120,"available":95098765312,"internally_reserved":12288,"allocated":95098777600,"data_stored":95098777600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2760495,"internal_metadata":0},"hb_peers":[0,1,3,5,6,7,8,9,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":16.838000000000001,"15min":6.0709999999999997},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.55200000000000005,"5min":728.28499999999997,"15min":728.28499999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.442,"5min":2.6040000000000001,"15min":1.179},"min":{"1min":0.28000000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68600000000000005,"5min":204.05199999999999,"15min":204.05199999999999},"last":0.40500000000000003}]},{"osd":1,"last update":"Sun Sep 4 14:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.504,"15min":0.48599999999999999},"min":{"1min":0.40100000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64600000000000002,"5min":0.747,"15min":0.747},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":3,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":10.555999999999999,"15min":20.555},"min":{"1min":0.37,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68100000000000005,"5min":501.30700000000002,"15min":501.30700000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":27.937999999999999,"5min":6.1120000000000001,"15min":2.52},"min":{"1min":0.41099999999999998,"5min":0.378,"15min":0.378},"max":{"1min":548.32399999999996,"5min":548.32399999999996,"15min":548.32399999999996},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":5,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":4.1059999999999999,"15min":1.962},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.72699999999999998,"5min":326.00200000000001,"15min":326.00200000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":50.252000000000002,"5min":19.597999999999999,"15min":7.1449999999999996},"min":{"1min":0.41899999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":958.46900000000005,"5min":958.46900000000005,"15min":958.46900000000005},"last":0.61699999999999999}]},{"osd":6,"last update":"Sun Sep 4 14:38:17 2022","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":1.2689999999999999,"15min":2.1160000000000001},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58699999999999997,"5min":39.411000000000001,"15min":39.411000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.439,"5min":18.437999999999999,"15min":26.834},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.67500000000000004,"5min":583.53800000000001,"15min":583.53800000000001},"last":0.46899999999999997}]},{"osd":7,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":12.416,"15min":24.251000000000001},"min":{"1min":0.38400000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.6619999999999999,"5min":681.09500000000003,"15min":681.09500000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":21.350999999999999,"5min":4.7939999999999996,"15min":2.1000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":417.06900000000002,"5min":417.06900000000002,"15min":417.06900000000002},"last":0.67700000000000005}]},{"osd":8,"last update":"Sun Sep 4 14:38:17 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60699999999999998,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72499999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":615.83699999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61399999999999999,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.746,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.63700000000000001}]},{"osd":9,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":4.1429999999999998,"15min":2.0270000000000001},"min":{"1min":0.42999999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71799999999999997,"5min":305.77300000000002,"15min":305.77300000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.629,"15min":0.65700000000000003},"min":{"1min":0.43099999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73299999999999998,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":0.65600000000000003}]},{"osd":12,"last update":"Sun Sep 4 14:38:17 2022","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":9.9550000000000001,"15min":3.7429999999999999},"min":{"1min":0.45100000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77200000000000002,"5min":934.52599999999995,"15min":934.52599999999995},"last":0.501},{"interface":"front","average":{"1min":0.62,"5min":6.3769999999999998,"15min":2.5569999999999999},"min":{"1min":0.42399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.745,"5min":547.75699999999995,"15min":547.75699999999995},"last":0.55200000000000005}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":15,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65400000000000003,"15min":0.68000000000000005},"min":{"1min":0.48799999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.74299999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":4.7880000000000003,"15min":2.1859999999999999},"min":{"1min":0.46800000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.76200000000000001,"5min":385.637,"15min":385.637},"last":0.751}]}]},{"osd":13,"up_from":0,"seq":1,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798820,"kb_used_data":92900596,"kb_used_omap":105,"kb_used_meta":0,"kb_avail":92900560,"statfs":{"total":95948165120,"available":95130173440,"internally_reserved":36864,"allocated":95130210304,"data_stored":95130210304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":108108,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":11,"up_from":21,"seq":90194313280,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":834292,"kb_used_data":92865100,"kb_used_omap":3154,"kb_used_meta":0,"kb_avail":92865088,"statfs":{"total":95948165120,"available":95093850112,"internally_reserved":12288,"allocated":95093862400,"data_stored":95093862400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3229712,"internal_metadata":0},"hb_peers":[0,1,2,3,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:53 2022","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.50900000000000001,"15min":0.66600000000000004},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.59799999999999998,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.72299999999999998,"15min":1.419},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67600000000000005,"5min":11.68,"15min":11.68},"last":0.48699999999999999}]},{"osd":1,"last update":"Sun Sep 4 14:38:19 2022","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":42.744999999999997,"15min":60.82},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68899999999999995,"5min":681.5,"15min":681.5},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":5.0839999999999996,"15min":2.056},"min":{"1min":0.34200000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67800000000000005,"5min":501.084,"15min":501.084},"last":0.61599999999999999}]},{"osd":2,"last update":"Sun Sep 4 14:38:37 2022","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73399999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35599999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.69299999999999995,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.67600000000000005}]},{"osd":3,"last update":"Sun Sep 4 14:38:53 2022","interfaces":[{"interface":"back","average":{"1min":45.515000000000001,"5min":17.567,"15min":6.202},"min":{"1min":0.27100000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":857.904,"5min":857.904,"15min":857.904},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.433,"15min":0.46200000000000002},"min":{"1min":0.29499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.65700000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":8,"last update":"Sun Sep 4 14:38:19 2022","interfaces":[{"interface":"back","average":{"1min":24.181000000000001,"5min":5.327,"15min":2.2320000000000002},"min":{"1min":0.41199999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":401.89800000000002,"5min":401.89800000000002,"15min":401.89800000000002},"last":0.501},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.75900000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57399999999999995}]},{"osd":9,"last update":"Sun Sep 4 14:38:53 2022","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.68999999999999995,"15min":1.0389999999999999},"min":{"1min":0.38200000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67600000000000005,"5min":12.435,"15min":12.435},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.627,"15min":0.80900000000000005},"min":{"1min":0.36199999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.748,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":0.435}]},{"osd":10,"last update":"Sun Sep 4 14:38:53 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":15.659000000000001,"15min":5.6529999999999996},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79000000000000004,"5min":981.94200000000001,"15min":981.94200000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":10.585000000000001,"15min":15.723000000000001},"min":{"1min":0.42099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71899999999999997,"5min":501.351,"15min":501.351},"last":0.71899999999999997}]},{"osd":12,"last update":"Sun Sep 4 14:38:09 2022","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.91900000000000004,"15min":1.24},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82399999999999995,"5min":10.442,"15min":10.442},"last":0.623},{"interface":"front","average":{"1min":21.632999999999999,"5min":12.137,"15min":4.4589999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":400.024,"5min":658.82100000000003,"15min":658.82100000000003},"last":0.58399999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":14,"last update":"Sun Sep 4 14:38:53 2022","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.64800000000000002,"15min":0.74399999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78900000000000003,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.621,"5min":1.6339999999999999,"15min":4.3479999999999999},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84099999999999997,"5min":86.528999999999996,"15min":86.528999999999996},"last":0.69299999999999995}]},{"osd":15,"last update":"Sun Sep 4 14:38:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":7.0700000000000003,"15min":2.863},"min":{"1min":0.40400000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.79700000000000004,"5min":775.30399999999997,"15min":775.30399999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":51.826999999999998,"5min":18.408000000000001,"15min":6.6500000000000004},"min":{"1min":0.42199999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":973.91099999999994,"5min":973.91099999999994,"15min":973.91099999999994},"last":0.52600000000000002}]}]},{"osd":10,"up_from":21,"seq":90194313280,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":817488,"kb_used_data":92881904,"kb_used_omap":1170,"kb_used_meta":0,"kb_avail":92881892,"statfs":{"total":95948165120,"available":95111057408,"internally_reserved":12288,"allocated":95111069696,"data_stored":95111069696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1198370,"internal_metadata":0},"hb_peers":[0,1,4,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":10.478999999999999,"15min":4.0780000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79800000000000004,"5min":787.36599999999999,"15min":787.36599999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.59999999999999998,"15min":0.70599999999999996},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72799999999999998,"5min":5.8550000000000004,"15min":5.8550000000000004},"last":0.5}]},{"osd":1,"last update":"Sun Sep 4 14:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":14.132999999999999,"15min":5.0700000000000003},"min":{"1min":0.35399999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72399999999999998,"5min":577.49599999999998,"15min":577.49599999999998},"last":0.64300000000000002},{"interface":"front","average":{"1min":47.359999999999999,"5min":17.821999999999999,"15min":6.3019999999999996},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":501.10000000000002,"5min":673.98199999999997,"15min":673.98199999999997},"last":0.59299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.69899999999999995,"15min":0.88900000000000001},"min":{"1min":0.35499999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.85299999999999998,"5min":8.6519999999999992,"15min":8.6519999999999992},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":2.399,"15min":1.411},"min":{"1min":0.34000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.751,"5min":146.78800000000001,"15min":146.78800000000001},"last":0.436}]},{"osd":5,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.61799999999999999,"15min":0.65700000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76500000000000001,"5min":1.26,"15min":1.26},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":1.8560000000000001,"15min":1.056},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.81499999999999995,"5min":100.83799999999999,"15min":100.83799999999999},"last":0.69099999999999995}]},{"osd":8,"last update":"Sun Sep 4 14:38:26 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.58399999999999996,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.68700000000000006,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52500000000000002,"15min":0.52600000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54400000000000004}]},{"osd":9,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56899999999999995,"15min":0.59899999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69799999999999995,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":16.117000000000001,"15min":31.686},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78800000000000003,"5min":806.59199999999998,"15min":806.59199999999998},"last":0.52800000000000002}]},{"osd":11,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66900000000000004,"15min":0.69899999999999995},"min":{"1min":0.503,"5min":0.438,"15min":0.438},"max":{"1min":0.81100000000000005,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64300000000000002,"15min":0.65900000000000003},"min":{"1min":0.47599999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78700000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57899999999999996}]},{"osd":12,"last update":"Sun Sep 4 14:38:26 2022","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.435,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.69899999999999995,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63400000000000001,"15min":0.65100000000000002},"min":{"1min":0.48299999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.81200000000000006,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.70099999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":14,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.749,"15min":0.98699999999999999},"min":{"1min":0.36799999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":9.7850000000000001,"15min":9.7850000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56599999999999995,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77100000000000002,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.51000000000000001}]},{"osd":15,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73399999999999999,"15min":0.83799999999999997},"min":{"1min":0.46600000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78500000000000003,"5min":3.6960000000000002,"15min":3.6960000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":25.239999999999998,"5min":5.585,"15min":2.3460000000000001},"min":{"1min":0.53400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":344.60599999999999,"5min":344.60599999999999,"15min":344.60599999999999},"last":0.71999999999999997}]}]},{"osd":9,"up_from":21,"seq":90194313280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":820612,"kb_used_data":92878768,"kb_used_omap":1674,"kb_used_meta":0,"kb_avail":92878768,"statfs":{"total":95948165120,"available":95107858432,"internally_reserved":0,"allocated":95107858432,"data_stored":95107858432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1714602,"internal_metadata":0},"hb_peers":[1,2,3,4,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.54400000000000004,"15min":0.74399999999999999},"min":{"1min":0.248,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.627,"5min":10.468999999999999,"15min":10.468999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.59099999999999997,"15min":0.83699999999999997},"min":{"1min":0.26400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51200000000000001,"5min":12.737,"15min":12.737},"last":0.54100000000000004}]},{"osd":2,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":2.2709999999999999,"15min":1.123},"min":{"1min":0.35599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0720000000000001,"5min":138.84800000000001,"15min":138.84800000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56000000000000005,"15min":0.54300000000000004},"min":{"1min":0.372,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.081,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.57899999999999996}]},{"osd":3,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":32.423999999999999,"5min":6.9119999999999999,"15min":2.6600000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":510.30700000000002,"5min":510.30700000000002,"15min":510.30700000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.56000000000000005,"5min":11.361000000000001,"15min":4.141},"min":{"1min":0.36099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.97199999999999998,"5min":975.76999999999998,"15min":975.76999999999998},"last":0.70299999999999996}]},{"osd":4,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.51100000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":3.383,"15min":1.4590000000000001},"min":{"1min":0.35199999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.65500000000000003,"5min":229.929,"15min":229.929},"last":0.56100000000000005}]},{"osd":8,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":3.3250000000000002,"15min":1.659},"min":{"1min":0.33000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.77900000000000003,"5min":226.983,"15min":226.983},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.70299999999999996,"15min":0.82799999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.96399999999999997,"5min":7.0890000000000004,"15min":7.0890000000000004},"last":0.72499999999999998}]},{"osd":10,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":21.329999999999998,"15min":37.470999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.91500000000000004,"5min":899.55499999999995,"15min":899.55499999999995},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":9.5800000000000001,"15min":9.2579999999999991},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80100000000000005,"5min":476.41399999999999,"15min":476.41399999999999},"last":0.85699999999999998}]},{"osd":11,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.38100000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98999999999999999,"5min":1.23,"15min":1.23},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":10.211,"15min":3.806},"min":{"1min":0.371,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2829999999999999,"5min":818.43600000000004,"15min":818.43600000000004},"last":0.79000000000000004}]},{"osd":12,"last update":"Sun Sep 4 14:38:37 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.374,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.434,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82899999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.78000000000000003}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":14,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0449999999999999,"5min":1.143,"15min":1.143},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65000000000000002,"15min":0.68000000000000005},"min":{"1min":0.44,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.98199999999999998,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.82099999999999995}]},{"osd":15,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":8.032,"15min":15.449},"min":{"1min":0.44600000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.054,"5min":389.73399999999998,"15min":389.73399999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63500000000000001,"15min":0.629},"min":{"1min":0.435,"5min":0.432,"15min":0.432},"max":{"1min":0.90300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.84299999999999997}]}]},{"osd":8,"up_from":21,"seq":90194313281,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":836784,"kb_used_data":92862608,"kb_used_omap":3341,"kb_used_meta":0,"kb_avail":92862596,"statfs":{"total":95948165120,"available":95091298304,"internally_reserved":12288,"allocated":95091310592,"data_stored":95091310592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3422126,"internal_metadata":0},"hb_peers":[0,1,2,3,4,7,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42999999999999999,"15min":0.50800000000000001},"min":{"1min":0.28599999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59999999999999998,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.42799999999999999,"15min":0.434},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52500000000000002}]},{"osd":1,"last update":"Sun Sep 4 14:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.65900000000000003},"min":{"1min":0.371,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67900000000000005,"5min":3.3450000000000002,"15min":3.3450000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.56899999999999995},"min":{"1min":0.45500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.47899999999999998}]},{"osd":2,"last update":"Sun Sep 4 14:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":1.8080000000000001,"15min":1.109},"min":{"1min":0.36899999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.77300000000000002,"5min":105.73099999999999,"15min":105.73099999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":10.58,"15min":4.0119999999999996},"min":{"1min":0.40600000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81599999999999995,"5min":713.197,"15min":713.197},"last":0.56299999999999994}]},{"osd":3,"last update":"Sun Sep 4 14:38:24 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61399999999999999,"15min":0.59399999999999997},"min":{"1min":0.46999999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90600000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.82599999999999996,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.57899999999999996}]},{"osd":4,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.48399999999999999},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":49.451000000000001,"5min":10.318,"15min":3.8079999999999998},"min":{"1min":0.36199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":783.87400000000002,"5min":783.87400000000002,"15min":783.87400000000002},"last":0.442}]},{"osd":7,"last update":"Sun Sep 4 14:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":11.683,"15min":41.109000000000002},"min":{"1min":0.497,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.93200000000000005,"5min":993.21799999999996,"15min":993.21799999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":8.8859999999999992,"15min":3.524},"min":{"1min":0.42999999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96799999999999997,"5min":943.09400000000005,"15min":943.09400000000005},"last":0.60399999999999998}]},{"osd":9,"last update":"Sun Sep 4 14:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63400000000000001,"15min":0.70199999999999996},"min":{"1min":0.39600000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.85999999999999999,"5min":2.859,"15min":2.859},"last":0.504},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.66000000000000003,"15min":0.75800000000000001},"min":{"1min":0.41799999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0169999999999999,"5min":3.3849999999999998,"15min":3.3849999999999998},"last":0.48999999999999999}]},{"osd":10,"last update":"Sun Sep 4 14:38:34 2022","interfaces":[{"interface":"back","average":{"1min":40.898000000000003,"5min":22.728999999999999,"15min":31.395},"min":{"1min":0.54000000000000004,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":644.41300000000001,"5min":667.33399999999995,"15min":667.33399999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66800000000000004,"15min":0.68100000000000005},"min":{"1min":0.499,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88300000000000001,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.627}]},{"osd":11,"last update":"Sun Sep 4 14:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":9.9139999999999997,"15min":6.4160000000000004},"min":{"1min":0.51800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98999999999999999,"5min":648.63699999999994,"15min":648.63699999999994},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.73299999999999998,"15min":0.748},"min":{"1min":0.59799999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.61699999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":15,"last update":"Sun Sep 4 14:38:54 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":2.5910000000000002,"15min":1.3899999999999999},"min":{"1min":0.60199999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.052,"5min":204.965,"15min":204.965},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.745,"5min":5.7880000000000003,"15min":2.5569999999999999},"min":{"1min":0.47899999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.081,"5min":479.31200000000001,"15min":479.31200000000001},"last":0.66100000000000003}]}]},{"osd":7,"up_from":21,"seq":90194313280,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":821156,"kb_used_data":92878236,"kb_used_omap":1534,"kb_used_meta":0,"kb_avail":92878224,"statfs":{"total":95948165120,"available":95107301376,"internally_reserved":12288,"allocated":95107313664,"data_stored":95107313664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1571569,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,8,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:37:56 2022","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":4.0780000000000003,"15min":1.722},"min":{"1min":0.35399999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.68600000000000005,"5min":375.97399999999999,"15min":375.97399999999999},"last":0.504},{"interface":"front","average":{"1min":37.834000000000003,"5min":23.681000000000001,"15min":34.439},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":784.19200000000001,"5min":784.19200000000001,"15min":784.19200000000001},"last":0.58099999999999996}]},{"osd":1,"last update":"Sun Sep 4 14:38:21 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":13.999000000000001,"15min":5.0579999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85999999999999999,"5min":929.38199999999995,"15min":929.38199999999995},"last":0.434},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79400000000000004,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.40699999999999997}]},{"osd":3,"last update":"Sun Sep 4 14:38:21 2022","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40799999999999997,"15min":0.42399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66900000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.31},{"interface":"front","average":{"1min":0.40699999999999997,"5min":12.471,"15min":4.4429999999999996},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.58299999999999996,"5min":946.95000000000005,"15min":946.95000000000005},"last":0.32400000000000001}]},{"osd":4,"last update":"Sun Sep 4 14:38:08 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.621,"15min":0.72799999999999998},"min":{"1min":0.38500000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71099999999999997,"5min":4.4770000000000003,"15min":4.4770000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":16.219999999999999,"15min":31.911999999999999},"min":{"1min":0.37,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69599999999999995,"5min":703.125,"15min":703.125},"last":0.61699999999999999}]},{"osd":5,"last update":"Sun Sep 4 14:38:41 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.55600000000000005,"15min":0.54200000000000004},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77800000000000002,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39100000000000001}]},{"osd":6,"last update":"Sun Sep 4 14:37:56 2022","interfaces":[{"interface":"back","average":{"1min":23.096,"5min":5.1210000000000004,"15min":2.1160000000000001},"min":{"1min":0.39500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":472.51600000000002,"5min":472.51600000000002,"15min":472.51600000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":5.2009999999999996,"15min":2.1429999999999998},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76300000000000001,"5min":598.846,"15min":598.846},"last":0.55800000000000005}]},{"osd":8,"last update":"Sun Sep 4 14:37:56 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":2.4420000000000002,"15min":1.4239999999999999},"min":{"1min":0.42299999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78800000000000003,"5min":227.75700000000001,"15min":227.75700000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.71699999999999997,"15min":0.80700000000000005},"min":{"1min":0.41499999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.876,"5min":4.5170000000000003,"15min":4.5170000000000003},"last":0.55200000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":11,"last update":"Sun Sep 4 14:37:56 2022","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59099999999999997,"15min":0.61199999999999999},"min":{"1min":0.33100000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.84799999999999998,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.441},{"interface":"front","average":{"1min":0.54800000000000004,"5min":17.202999999999999,"15min":6.1630000000000003},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82799999999999996,"5min":845.21000000000004,"15min":845.21000000000004},"last":0.377}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":14,"last update":"Sun Sep 4 14:37:56 2022","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.83799999999999997,"15min":1.004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.90600000000000003,"5min":7.9539999999999997,"15min":7.9539999999999997},"last":0.628},{"interface":"front","average":{"1min":0.64600000000000002,"5min":5.7060000000000004,"15min":2.3860000000000001},"min":{"1min":0.47699999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.85099999999999998,"5min":403.21699999999998,"15min":403.21699999999998},"last":0.59499999999999997}]},{"osd":15,"last update":"Sun Sep 4 14:37:56 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":11.553000000000001,"15min":4.3200000000000003},"min":{"1min":0.37,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81399999999999995,"5min":501.173,"15min":501.173},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":15.565,"15min":30.481000000000002},"min":{"1min":0.378,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94199999999999995,"5min":708.476,"15min":708.476},"last":0.46400000000000002}]}]},{"osd":6,"up_from":21,"seq":90194313280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":828076,"kb_used_data":92871316,"kb_used_omap":1449,"kb_used_meta":0,"kb_avail":92871304,"statfs":{"total":95948165120,"available":95100215296,"internally_reserved":12288,"allocated":95100227584,"data_stored":95100227584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1484544,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:02 2022","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.52400000000000002,"15min":0.503},"min":{"1min":0.377,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.876,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.41699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76800000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.437}]},{"osd":1,"last update":"Sun Sep 4 14:37:51 2022","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.94799999999999995,"15min":1.353},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":18.829999999999998,"15min":18.829999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.93999999999999995,"15min":1.3140000000000001},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80000000000000004,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.67800000000000005}]},{"osd":2,"last update":"Sun Sep 4 14:38:20 2022","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":7.508,"15min":2.8260000000000001},"min":{"1min":0.314,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80000000000000004,"5min":775.024,"15min":775.024},"last":0.56499999999999995},{"interface":"front","average":{"1min":45.057000000000002,"5min":9.3740000000000006,"15min":3.4420000000000002},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":892.12300000000005,"5min":892.12300000000005,"15min":892.12300000000005},"last":0.54100000000000004}]},{"osd":3,"last update":"Sun Sep 4 14:37:51 2022","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":7.9930000000000003,"15min":3.3279999999999998},"min":{"1min":0.44800000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.91300000000000003,"5min":648.39700000000005,"15min":648.39700000000005},"last":0.69099999999999995},{"interface":"front","average":{"1min":9.0600000000000005,"5min":10.468999999999999,"15min":4.0129999999999999},"min":{"1min":0.502,"5min":0.317,"15min":0.317},"max":{"1min":167.96899999999999,"5min":728.35400000000004,"15min":728.35400000000004},"last":0.71499999999999997}]},{"osd":4,"last update":"Sun Sep 4 14:38:20 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":4.085,"15min":1.7350000000000001},"min":{"1min":0.40500000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89000000000000001,"5min":387.75799999999998,"15min":387.75799999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":19.766999999999999,"15min":31.701000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.90900000000000003,"5min":504.84899999999999,"15min":504.84899999999999},"last":0.53000000000000003}]},{"osd":5,"last update":"Sun Sep 4 14:37:51 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.94399999999999995,"15min":1.288},"min":{"1min":0.442,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96599999999999997,"5min":16.707999999999998,"15min":16.707999999999998},"last":0.69999999999999996},{"interface":"front","average":{"1min":39.378999999999998,"5min":8.3379999999999992,"15min":3.1539999999999999},"min":{"1min":0.46100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":737.37,"5min":737.37,"15min":737.37},"last":0.77400000000000002}]},{"osd":7,"last update":"Sun Sep 4 14:37:51 2022","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.72099999999999997,"15min":0.76000000000000001},"min":{"1min":0.50900000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.98199999999999998,"5min":2.996,"15min":2.996},"last":0.60299999999999998},{"interface":"front","average":{"1min":10.425000000000001,"5min":25.491,"15min":33.902000000000001},"min":{"1min":0.55000000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":194.964,"5min":711.51199999999994,"15min":711.51199999999994},"last":0.55000000000000004}]},{"osd":8,"last update":"Sun Sep 4 14:38:20 2022","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":11.265000000000001,"15min":21.899000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89000000000000001,"5min":531.23900000000003,"15min":531.23900000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92400000000000004,"5min":1.012,"15min":1.012},"last":0.68500000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":10,"last update":"Sun Sep 4 14:37:51 2022","interfaces":[{"interface":"back","average":{"1min":27.594000000000001,"5min":13.659000000000001,"15min":4.9429999999999996},"min":{"1min":0.42899999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":540.61099999999999,"5min":540.61099999999999,"15min":540.61099999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":5.2290000000000001,"15min":2.2549999999999999},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.93100000000000005,"5min":500.86099999999999,"15min":500.86099999999999},"last":0.57699999999999996}]},{"osd":11,"last update":"Sun Sep 4 14:37:51 2022","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.69399999999999995,"15min":0.68999999999999995},"min":{"1min":0.53500000000000003,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.93999999999999995,"5min":0.997,"15min":0.997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69399999999999995,"15min":0.69099999999999995},"min":{"1min":0.47499999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.98899999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.69199999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":14,"last update":"Sun Sep 4 14:37:51 2022","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":21.559999999999999,"15min":42.450000000000003},"min":{"1min":0.49299999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.075,"5min":993.29899999999998,"15min":993.29899999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68999999999999995,"5min":15.4,"15min":30.140000000000001},"min":{"1min":0.50700000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0329999999999999,"5min":697.30399999999997,"15min":697.30399999999997},"last":0.66400000000000003}]},{"osd":15,"last update":"Sun Sep 4 14:38:20 2022","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":3.3500000000000001,"15min":1.627},"min":{"1min":0.52500000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.0169999999999999,"5min":274.17399999999998,"15min":274.17399999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":11.914999999999999,"5min":2.9860000000000002,"15min":1.528},"min":{"1min":0.56100000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":223.04400000000001,"5min":223.04400000000001,"15min":223.04400000000001},"last":0.72899999999999998}]}]},{"osd":1,"up_from":21,"seq":90194313279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":835988,"kb_used_data":92863404,"kb_used_omap":2166,"kb_used_meta":0,"kb_avail":92863392,"statfs":{"total":95948165120,"available":95092113408,"internally_reserved":12288,"allocated":95092125696,"data_stored":95092125696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2218097,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7,8,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.47299999999999998,"15min":0.499},"min":{"1min":0.29599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67300000000000004,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.54400000000000004,"15min":0.63100000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64000000000000001,"5min":3.839,"15min":3.839},"last":0.63900000000000001}]},{"osd":2,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82699999999999996,"5min":1.137,"15min":1.137},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.38,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76300000000000001,"5min":0.995,"15min":0.995},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66700000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55200000000000005,"15min":0.56799999999999995},"min":{"1min":0.40600000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72199999999999998,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53500000000000003}]},{"osd":4,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.60999999999999999,"15min":0.70299999999999996},"min":{"1min":0.371,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.63200000000000001,"5min":3.996,"15min":3.996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":6.8140000000000001,"15min":13.076000000000001},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.74099999999999999,"5min":308.714,"15min":308.714},"last":0.67100000000000004}]},{"osd":5,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":6.9189999999999996,"15min":2.6059999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59799999999999998,"5min":583.06399999999996,"15min":583.06399999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.443,"15min":0.441},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.72899999999999998,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50900000000000001}]},{"osd":6,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.40699999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86099999999999999,"5min":1.173,"15min":1.173},"last":0.74299999999999999},{"interface":"front","average":{"1min":43.801000000000002,"5min":9.2010000000000005,"15min":3.4540000000000002},"min":{"1min":0.39400000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":865.29600000000005,"5min":865.29600000000005,"15min":865.29600000000005},"last":0.751}]},{"osd":7,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":45.802,"15min":85.352999999999994},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78000000000000003,"5min":906.88800000000003,"15min":906.88800000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.48199999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81200000000000006,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.66200000000000003}]},{"osd":8,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":33.216999999999999,"5min":22.084,"15min":32.747999999999998},"min":{"1min":0.374,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":500.97800000000001,"5min":501.06900000000002,"15min":501.06900000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":3.2770000000000001,"15min":5.9660000000000002},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.81899999999999995,"5min":133.97300000000001,"15min":133.97300000000001},"last":0.72899999999999998}]},{"osd":11,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59999999999999998,"15min":0.63},"min":{"1min":0.45800000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.77300000000000002,"5min":1.55,"15min":1.55},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":2.0750000000000002,"15min":3.5379999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79200000000000004,"5min":69.060000000000002,"15min":69.060000000000002},"last":0.63100000000000001}]},{"osd":14,"last update":"Sun Sep 4 14:38:04 2022","interfaces":[{"interface":"back","average":{"1min":47.353999999999999,"5min":10.023999999999999,"15min":3.8300000000000001},"min":{"1min":0.46400000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":934.702,"5min":934.702,"15min":934.702},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":9.0440000000000005,"15min":3.4660000000000002},"min":{"1min":0.47899999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96899999999999997,"5min":714.40800000000002,"15min":714.40800000000002},"last":0.78800000000000003}]}]},{"osd":0,"up_from":21,"seq":90194313281,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":837136,"kb_used_data":92862256,"kb_used_omap":3501,"kb_used_meta":0,"kb_avail":92862244,"statfs":{"total":95948165120,"available":95090937856,"internally_reserved":12288,"allocated":95090950144,"data_stored":95090950144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3585352,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":3.8410000000000002,"15min":2.2010000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.81699999999999995,"5min":344.70699999999999,"15min":344.70699999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":3.9569999999999999,"15min":2.4119999999999999},"min":{"1min":0.38800000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.93100000000000005,"5min":344.47300000000001,"15min":344.47300000000001},"last":0.74199999999999999}]},{"osd":2,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.51400000000000001,"15min":0.53700000000000003},"min":{"1min":0.38800000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71399999999999997,"5min":1.343,"15min":1.343},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":6.5880000000000001,"15min":2.6680000000000001},"min":{"1min":0.44500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.75800000000000001,"5min":687.74099999999999,"15min":687.74099999999999},"last":0.82799999999999996}]},{"osd":3,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":39.343000000000004,"5min":8.3300000000000001,"15min":3.1619999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":891.15599999999995,"5min":891.15599999999995,"15min":891.15599999999995},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":8.5329999999999995,"15min":3.2290000000000001},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76000000000000001,"5min":919.09100000000001,"15min":919.09100000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Sun Sep 4 14:38:39 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42299999999999999,"15min":0.41199999999999998},"min":{"1min":0.29999999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.67800000000000005,"5min":2.141,"15min":2.141},"last":0.39900000000000002},{"interface":"front","average":{"1min":3.1579999999999999,"5min":0.95799999999999996,"15min":0.60199999999999998},"min":{"1min":0.28499999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":51.243000000000002,"5min":51.243000000000002,"15min":51.243000000000002},"last":0.42899999999999999}]},{"osd":5,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56699999999999995,"15min":0.62},"min":{"1min":0.35199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.89900000000000002,"5min":3.6589999999999998,"15min":3.6589999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.91900000000000004,"15min":1.2569999999999999},"min":{"1min":0.432,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.94799999999999995,"5min":18.632999999999999,"15min":18.632999999999999},"last":0.75}]},{"osd":6,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.64500000000000002,"15min":0.67400000000000004},"min":{"1min":0.38600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79200000000000004,"5min":4.6589999999999998,"15min":4.6589999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63600000000000001,"15min":0.67800000000000005},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90600000000000003,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":0.78600000000000003}]},{"osd":7,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.39600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.82499999999999996,"5min":1.387,"15min":1.387},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":17.045000000000002,"15min":30.035},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.91900000000000004,"5min":752.76300000000003,"15min":752.76300000000003},"last":0.75800000000000001}]},{"osd":8,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65200000000000002,"15min":0.69999999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.93999999999999995,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":27.652000000000001,"15min":41.110999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.80400000000000005,"5min":992.69399999999996,"15min":992.69399999999996},"last":0.61699999999999999}]},{"osd":9,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65200000000000002,"15min":0.70799999999999996},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.92000000000000004,"5min":3.6819999999999999,"15min":3.6819999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":7.9889999999999999,"5min":2.1509999999999998,"15min":1.22},"min":{"1min":0.49399999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":168.52500000000001,"5min":168.52500000000001,"15min":168.52500000000001},"last":0.77000000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":15,"last update":"Sun Sep 4 14:37:58 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":6.726,"15min":2.6779999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.75800000000000001,"5min":476.10899999999998,"15min":476.10899999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.69999999999999996,"15min":0.75},"min":{"1min":0.48199999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93999999999999995,"5min":3.7469999999999999,"15min":3.7469999999999999},"last":0.88300000000000001}]}]},{"osd":2,"up_from":21,"seq":90194313280,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":820080,"kb_used_data":92879312,"kb_used_omap":1308,"kb_used_meta":0,"kb_avail":92879300,"statfs":{"total":95948165120,"available":95108403200,"internally_reserved":12288,"allocated":95108415488,"data_stored":95108415488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1339881,"internal_metadata":0},"hb_peers":[0,1,3,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:37:54 2022","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.75900000000000001,"15min":1.0209999999999999},"min":{"1min":0.37,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.755,"5min":12.19,"15min":12.19},"last":0.745},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.0289999999999999,"15min":1.5389999999999999},"min":{"1min":0.39000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85199999999999998,"5min":23.140000000000001,"15min":23.140000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Sun Sep 4 14:37:54 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":7.665,"15min":3.5019999999999998},"min":{"1min":0.33600000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79800000000000004,"5min":711.13499999999999,"15min":711.13499999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":5.9699999999999998,"15min":11.359},"min":{"1min":0.36599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93000000000000005,"5min":226.15000000000001,"15min":226.15000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Sun Sep 4 14:37:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.766,"15min":2.9060000000000001},"min":{"1min":0.47599999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.97799999999999998,"5min":44.57,"15min":44.57},"last":0.626},{"interface":"front","average":{"1min":0.629,"5min":0.59299999999999997,"15min":0.58299999999999996},"min":{"1min":0.435,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.57899999999999996}]},{"osd":4,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":7.0090000000000003,"15min":2.7000000000000002},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77800000000000002,"5min":648.851,"15min":648.851},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":2.5539999999999998,"15min":1.2430000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.83199999999999996,"5min":188.68199999999999,"15min":188.68199999999999},"last":0.77700000000000002}]},{"osd":5,"last update":"Sun Sep 4 14:38:29 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":3.6000000000000001,"15min":1.6240000000000001},"min":{"1min":0.379,"5min":0.315,"15min":0.315},"max":{"1min":0.86899999999999999,"5min":271.64299999999997,"15min":271.64299999999997},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":7.9290000000000003,"15min":3.069},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.74099999999999999,"5min":659.28899999999999,"15min":659.28899999999999},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":7,"last update":"Sun Sep 4 14:37:54 2022","interfaces":[{"interface":"back","average":{"1min":14.654,"5min":6.8310000000000004,"15min":2.6869999999999998},"min":{"1min":0.45800000000000002,"5min":0.375,"15min":0.375},"max":{"1min":280.91399999999999,"5min":280.91399999999999,"15min":280.91399999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.65800000000000003,"15min":0.73199999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76300000000000001,"5min":3.319,"15min":3.319},"last":0.52200000000000002}]},{"osd":8,"last update":"Sun Sep 4 14:37:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.69899999999999995,"15min":0.755},"min":{"1min":0.47899999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88900000000000001,"5min":3.343,"15min":3.343},"last":0.78500000000000003},{"interface":"front","average":{"1min":21.390999999999998,"5min":4.7619999999999996,"15min":2},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":415.916,"5min":415.916,"15min":415.916},"last":0.83299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":10,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":19.405000000000001,"15min":6.8780000000000001},"min":{"1min":0.36699999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78300000000000003,"5min":851.88499999999999,"15min":851.88499999999999},"last":0.435},{"interface":"front","average":{"1min":0.59599999999999997,"5min":5.8639999999999999,"15min":2.3540000000000001},"min":{"1min":0.40000000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":500.99299999999999,"15min":500.99299999999999},"last":0.48299999999999998}]},{"osd":11,"last update":"Sun Sep 4 14:38:19 2022","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":15.121,"15min":5.4809999999999999},"min":{"1min":0.432,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95799999999999996,"5min":838.76400000000001,"15min":838.76400000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":50.322000000000003,"5min":16.082000000000001,"15min":5.8179999999999996},"min":{"1min":0.505,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":633.26700000000005,"5min":633.26700000000005,"15min":633.26700000000005},"last":0.59899999999999998}]},{"osd":12,"last update":"Sun Sep 4 14:38:02 2022","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.45100000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0289999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":21.579999999999998,"15min":42.488999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.454,"5min":993.596,"15min":993.596},"last":0.84399999999999997}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":14,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":9.4570000000000007,"15min":3.621},"min":{"1min":0.42299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.92500000000000004,"5min":501.26400000000001,"15min":501.26400000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68799999999999994,"15min":0.68999999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94099999999999995,"5min":1.502,"15min":1.502},"last":0.66700000000000004}]},{"osd":15,"last update":"Sun Sep 4 14:38:19 2022","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":6.133,"15min":2.5430000000000001},"min":{"1min":0.61099999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.0129999999999999,"5min":491.33999999999997,"15min":491.33999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.746,"15min":0.746},"min":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0169999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.71199999999999997}]}]},{"osd":3,"up_from":21,"seq":90194313280,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":846292,"kb_used_data":92853100,"kb_used_omap":3186,"kb_used_meta":0,"kb_avail":92853088,"statfs":{"total":95948165120,"available":95081562112,"internally_reserved":12288,"allocated":95081574400,"data_stored":95081574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3263120,"internal_metadata":0},"hb_peers":[1,2,4,5,6,7,8,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":1,"last update":"Sun Sep 4 14:38:19 2022","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":11.587999999999999,"15min":6.0679999999999996},"min":{"1min":0.32300000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77600000000000002,"5min":895.83600000000001,"15min":895.83600000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":4.085,"15min":7.6340000000000003},"min":{"1min":0.34599999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79800000000000004,"5min":174.59399999999999,"15min":174.59399999999999},"last":0.39900000000000002}]},{"osd":2,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":15.449,"15min":11.465},"min":{"1min":0.41599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.871,"5min":612.41800000000001,"15min":612.41800000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61799999999999999,"15min":0.67200000000000004},"min":{"1min":0.435,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.71199999999999997,"5min":4.4980000000000002,"15min":4.4980000000000002},"last":0.54400000000000004}]},{"osd":4,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51000000000000001,"15min":0.52500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.58299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.501},"min":{"1min":0.317,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.48799999999999999}]},{"osd":5,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":11.9,"15min":23.274000000000001},"min":{"1min":0.373,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.746,"5min":568.99599999999998,"15min":568.99599999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":55.006,"5min":23.931000000000001,"15min":8.3190000000000008},"min":{"1min":0.41099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":903.36000000000001,"5min":947.37099999999998,"15min":947.37099999999998},"last":0.436}]},{"osd":6,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":5.069,"15min":2.2130000000000001},"min":{"1min":0.35299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78700000000000003,"5min":353.42000000000002,"15min":353.42000000000002},"last":0.63},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64500000000000002}]},{"osd":7,"last update":"Sun Sep 4 14:38:02 2022","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.57699999999999996,"15min":0.67700000000000005},"min":{"1min":0.28599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.67500000000000004,"5min":4.2969999999999997,"15min":4.2969999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.49099999999999999,"15min":0.53300000000000003},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.51300000000000001,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.38}]},{"osd":8,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.39800000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":3.1259999999999999,"15min":1.5960000000000001},"min":{"1min":0.40000000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.84399999999999997,"5min":146.69800000000001,"15min":146.69800000000001},"last":0.52100000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":10,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.878,"15min":1.1100000000000001},"min":{"1min":0.48799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.878,"5min":7.6959999999999997,"15min":7.6959999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":11.523999999999999,"15min":4.2720000000000002},"min":{"1min":0.36899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0149999999999999,"5min":928.69299999999998,"15min":928.69299999999998},"last":0.53800000000000003}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":14,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.488,"15min":1.0640000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96499999999999997,"5min":75.988,"15min":75.988},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.75700000000000001,"15min":0.83699999999999997},"min":{"1min":0.42899999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93400000000000005,"5min":4.54,"15min":4.54},"last":0.63800000000000001}]},{"osd":15,"last update":"Sun Sep 4 14:38:19 2022","interfaces":[{"interface":"back","average":{"1min":46.110999999999997,"5min":18.309999999999999,"15min":6.4969999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":956.97500000000002,"5min":956.97500000000002,"15min":956.97500000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.72499999999999998,"15min":0.81000000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94799999999999995,"5min":4.5250000000000004,"15min":4.5250000000000004},"last":0.621}]}]},{"osd":4,"up_from":21,"seq":90194313279,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":832072,"kb_used_data":92867320,"kb_used_omap":2508,"kb_used_meta":0,"kb_avail":92867308,"statfs":{"total":95948165120,"available":95096123392,"internally_reserved":12288,"allocated":95096135680,"data_stored":95096135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2569199,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:06 2022","interfaces":[{"interface":"back","average":{"1min":48.829000000000001,"5min":10.130000000000001,"15min":3.6779999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":919.59500000000003,"5min":919.59500000000003,"15min":919.59500000000003},"last":0.624},{"interface":"front","average":{"1min":0.52800000000000002,"5min":12.130000000000001,"15min":8.1219999999999999},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.91000000000000003,"5min":844.32600000000002,"15min":844.32600000000002},"last":0.73699999999999999}]},{"osd":1,"last update":"Sun Sep 4 14:38:18 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.86899999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":24.282,"5min":13.726000000000001,"15min":4.9199999999999999},"min":{"1min":0.35299999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":403.02300000000002,"5min":922.62400000000002,"15min":922.62400000000002},"last":0.55100000000000005}]},{"osd":2,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":6.7539999999999996,"15min":2.8050000000000002},"min":{"1min":0.43099999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88900000000000001,"5min":540.26400000000001,"15min":540.26400000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.39100000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.74199999999999999,"5min":1.226,"15min":1.226},"last":0.751}]},{"osd":3,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67100000000000004,"15min":0.67300000000000004},"min":{"1min":0.439,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.77900000000000003,"15min":0.85399999999999998},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82899999999999996,"5min":5.5339999999999998,"15min":5.5339999999999998},"last":0.57699999999999996}]},{"osd":5,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":3.5910000000000002,"5min":1.1950000000000001,"15min":0.80100000000000005},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":58.374000000000002,"5min":58.374000000000002,"15min":58.374000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65000000000000002,"15min":0.64800000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.79900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65900000000000003}]},{"osd":6,"last update":"Sun Sep 4 14:38:18 2022","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":6.8220000000000001,"15min":2.6890000000000001},"min":{"1min":0.46500000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89600000000000002,"5min":672.35400000000004,"15min":672.35400000000004},"last":0.72199999999999998},{"interface":"front","average":{"1min":32.981999999999999,"5min":7.133,"15min":2.7989999999999999},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":549.25999999999999,"5min":549.25999999999999,"15min":549.25999999999999},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":8,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":7.9509999999999996,"15min":3.0659999999999998},"min":{"1min":0.47599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86899999999999999,"5min":514.50300000000004,"15min":514.50300000000004},"last":0.77600000000000002},{"interface":"front","average":{"1min":35.395000000000003,"5min":10.94,"15min":4.1890000000000001},"min":{"1min":0.40300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":661.00900000000001,"5min":661.00900000000001,"15min":661.00900000000001},"last":0.94999999999999996}]},{"osd":9,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":11.747999999999999,"15min":7.04},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.234,"5min":501.34199999999998,"15min":501.34199999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.747,"5min":5.9749999999999996,"15min":2.4590000000000001},"min":{"1min":0.47999999999999998,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0369999999999999,"5min":501.22199999999998,"15min":501.22199999999998},"last":0.78800000000000003}]},{"osd":10,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":3.2240000000000002,"15min":1.5489999999999999},"min":{"1min":0.48699999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.2549999999999999,"5min":227.93700000000001,"15min":227.93700000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.751,"15min":0.73499999999999999},"min":{"1min":0.60399999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.98599999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.80500000000000005}]},{"osd":11,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":22.565999999999999,"5min":5.2610000000000001,"15min":2.4089999999999998},"min":{"1min":0.55200000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":414.88200000000001,"5min":414.88200000000001,"15min":414.88200000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":5.0830000000000002,"15min":2.1909999999999998},"min":{"1min":0.47899999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.083,"5min":436.26900000000001,"15min":436.26900000000001},"last":0.93799999999999994}]},{"osd":12,"last update":"Sun Sep 4 14:38:18 2022","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.754,"15min":0.73399999999999999},"min":{"1min":0.496,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.78900000000000003,"15min":0.73199999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.097,"5min":3.7200000000000002,"15min":3.7200000000000002},"last":0.97499999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":14,"last update":"Sun Sep 4 14:38:18 2022","interfaces":[{"interface":"back","average":{"1min":46.649000000000001,"5min":17.466000000000001,"15min":6.3220000000000001},"min":{"1min":0.68200000000000005,"5min":0.505,"15min":0.505},"max":{"1min":779.50199999999995,"5min":779.50199999999995,"15min":779.50199999999995},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":3.238,"15min":1.5940000000000001},"min":{"1min":0.59999999999999998,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0509999999999999,"5min":272.31200000000001,"15min":272.31200000000001},"last":0.96399999999999997}]},{"osd":15,"last update":"Sun Sep 4 14:37:59 2022","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86399999999999999,"15min":0.81699999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.139,"5min":3.931,"15min":3.931},"last":1.026},{"interface":"front","average":{"1min":40.223999999999997,"5min":8.8460000000000001,"15min":3.6480000000000001},"min":{"1min":0.56899999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":501.55500000000001,"5min":501.55500000000001,"15min":501.55500000000001},"last":1.0349999999999999}]}]},{"osd":5,"up_from":21,"seq":90194313280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":817844,"kb_used_data":92881548,"kb_used_omap":1544,"kb_used_meta":0,"kb_avail":92881536,"statfs":{"total":95948165120,"available":95110692864,"internally_reserved":12288,"allocated":95110705152,"data_stored":95110705152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1581898,"internal_metadata":0},"hb_peers":[0,2,3,4,6,7,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":7.2199999999999998,"15min":2.681},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.54800000000000004,"5min":611.73599999999999,"15min":611.73599999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.44400000000000001,"15min":0.42599999999999999},"min":{"1min":0.28299999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Sun Sep 4 14:38:03 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.875,"5min":0.876,"15min":0.876},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":3.8980000000000001,"15min":1.6679999999999999},"min":{"1min":0.40100000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.1259999999999999,"5min":366.96899999999999,"15min":366.96899999999999},"last":0.74099999999999999}]},{"osd":3,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.312},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.503,"15min":0.48999999999999999},"min":{"1min":0.374,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.371}]},{"osd":4,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":6.2140000000000004,"15min":2.4009999999999998},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64800000000000002,"5min":336.06599999999997,"15min":336.06599999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":18.728000000000002,"5min":4.1479999999999997,"15min":1.7290000000000001},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":347.00200000000001,"5min":347.00200000000001,"15min":347.00200000000001},"last":0.48499999999999999}]},{"osd":6,"last update":"Sun Sep 4 14:37:56 2022","interfaces":[{"interface":"back","average":{"1min":26.925000000000001,"5min":5.8310000000000004,"15min":2.3199999999999998},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":501.06700000000001,"5min":501.06700000000001,"15min":501.06700000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":26.925999999999998,"5min":6.508,"15min":2.5830000000000002},"min":{"1min":0.43099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":500.983,"5min":500.983,"15min":500.983},"last":0.76100000000000001}]},{"osd":7,"last update":"Sun Sep 4 14:38:23 2022","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":10.023999999999999,"15min":3.706},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.93000000000000005,"5min":852.12099999999998,"15min":852.12099999999998},"last":0.435},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54100000000000004,"15min":0.52900000000000003},"min":{"1min":0.42099999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51100000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":10,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":10.148999999999999,"15min":3.8010000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77300000000000002,"5min":859.45000000000005,"15min":859.45000000000005},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":6.3129999999999997,"15min":11.977},"min":{"1min":0.47299999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.91200000000000003,"5min":269.21800000000002,"15min":269.21800000000002},"last":0.77700000000000002}]},{"osd":11,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":10.849,"15min":4.0449999999999999},"min":{"1min":0.373,"5min":0.371,"15min":0.371},"max":{"1min":0.86299999999999999,"5min":921.39499999999998,"15min":921.39499999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.41799999999999998}]},{"osd":12,"last update":"Sun Sep 4 14:38:00 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":17.286000000000001,"15min":34.011000000000003},"min":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81499999999999995,"5min":711.00999999999999,"15min":711.00999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76600000000000001,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.53300000000000003}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":14,"last update":"Sun Sep 4 14:37:55 2022","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":27.789000000000001,"15min":46.134999999999998},"min":{"1min":0.41799999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":749.64999999999998,"15min":749.64999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":5.0250000000000004,"15min":2.1280000000000001},"min":{"1min":0.501,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.94399999999999995,"5min":479.36799999999999,"15min":479.36799999999999},"last":0.66400000000000003}]}]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2022-09-04T14:38:56.654 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:56.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.723448+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:56.683 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:38:56.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.733327+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:56.713 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.749505+0000", , 2022-09-04T14:38:56.713 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 283.88904326099998, , 2022-09-04T14:38:56.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.036741401, Ignoring malparsed backtrace: 2022-09-04T14:38:56.744 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.759235+0000", , 2022-09-04T14:38:56.744 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 283.87931320400003, , 2022-09-04T14:38:56.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.026081963, Ignoring malparsed backtrace: 2022-09-04T14:38:56.767 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:38:56.767 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:38:56.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:56.791 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:56.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", Ignoring malparsed backtrace: 2022-09-04T14:38:56.814 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:109 3.as0 3:5170752b:::smithi16339483-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:38:56.814 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.799999+0000", , 2022-09-04T14:38:56.815 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 283.838549108, Ignoring malparsed backtrace: 2022-09-04T14:38:56.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:56.838 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:38:56.838 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:56.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:56.861 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:38:56.861 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030874522000000001 , 2022-09-04T14:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.014915832 , 2022-09-04T14:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0057189500000000004 , 2022-09-04T14:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8559000000000005e-05 , 2022-09-04T14:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:56.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.091533+0000", , 2022-09-04T14:38:56.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9589999999999997e-06 , 2022-09-04T14:38:56.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:57.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:57.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:14.246847+0000", , 2022-09-04T14:38:57.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6739999999999996e-06 Ignoring malparsed backtrace: 2022-09-04T14:38:57.507 INFO:teuthology.orchestra.run.smithi155.stdout:{ , 2022-09-04T14:38:57.507 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, , 2022-09-04T14:38:57.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, Ignoring malparsed backtrace: 2022-09-04T14:38:57.532 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:57.532 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:38:57.532 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:57.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:57.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:57.556 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:57.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:57.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:57.580 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:57.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1722999999999999e-05 , 2022-09-04T14:38:57.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:57.605 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:57.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00095956199999999996 , 2022-09-04T14:38:57.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:57.628 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:57.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001069609 , 2022-09-04T14:38:57.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:57.653 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:57.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0180000000000004e-06 , 2022-09-04T14:38:57.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:57.678 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:57.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6855999999999999e-05 , 2022-09-04T14:38:57.703 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:57.703 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:38:57.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783673+0000", , 2022-09-04T14:38:57.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.999999 , 2022-09-04T14:38:57.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:57.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784665+0000", , 2022-09-04T14:38:57.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8630000000000008e-06 , 2022-09-04T14:38:57.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:57.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.794534+0000", , 2022-09-04T14:38:57.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0093767590000000001 , 2022-09-04T14:38:57.775 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:38:57.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:57.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.830012+0000", , 2022-09-04T14:38:57.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:57.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:38:57.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.905044+0000", , 2022-09-04T14:38:57.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:57.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:38:57.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:56.032035+0000", , 2022-09-04T14:38:57.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:38:57.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:38:57.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.175821+0000", , 2022-09-04T14:38:57.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017596199999999999 Ignoring malparsed backtrace: 2022-09-04T14:38:57.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", , 2022-09-04T14:38:57.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:21.272124+0000", , 2022-09-04T14:38:57.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00323456 Ignoring malparsed backtrace: 2022-09-04T14:38:58.225 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], , 2022-09-04T14:38:58.226 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, , 2022-09-04T14:38:58.226 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:58.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0066417179999999996 , 2022-09-04T14:38:58.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:58.394 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:58.429 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 157 , 2022-09-04T14:38:58.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:58.430 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:38:58.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858431+0000", , 2022-09-04T14:38:58.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:38:58.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:58.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801919+0000", , 2022-09-04T14:38:58.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.838e-05 , 2022-09-04T14:38:58.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:58.509 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.816182+0000", , 2022-09-04T14:38:58.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1827e-05 , 2022-09-04T14:38:58.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:38:58.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:38:58.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.841089+0000", , 2022-09-04T14:38:58.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:38:58.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:58.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.847619+0000", , 2022-09-04T14:38:58.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2782999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:38:58.589 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:38:58.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:58.589 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:38:58.616 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:58.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:38:58.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949072+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:58.641 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:58.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:38:58.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.954373+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:58.665 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:58.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:38:58.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.957384+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:58.693 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:58.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", , 2022-09-04T14:38:58.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.957504+0000", Ignoring malparsed backtrace: 2022-09-04T14:38:58.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:58.721 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:58.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:38:58.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:58.750 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:58.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:38:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:38:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:38:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:38:58.812 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} , 2022-09-04T14:38:58.812 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} , 2022-09-04T14:38:58.812 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} Ignoring malparsed backtrace: 2022-09-04T14:39:00.635+0000 7efdf07f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38715) UID: 0 , 2022-09-04T14:39:00.736+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 , 2022-09-04T14:39:00.739 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:39:01.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:01.233 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:01.233 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.7s3 47/33 ECSubWrite(tid=80, reqid=client.4557.0:287, at_version=0'0, trim_to=0'0, roll_forward_to=47'25))", Ignoring malparsed backtrace: 2022-09-04T14:39:01.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:01.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.842593+0000", , 2022-09-04T14:39:01.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6279000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:01.284 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:01.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:01.285 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:01.316 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:01.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:01.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682584+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:01.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910693+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:39:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:39:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", , 2022-09-04T14:39:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:39:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:01.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8249999999999999e-06 , 2022-09-04T14:39:01.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:01.469 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:01.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3364000000000001e-05 , 2022-09-04T14:39:01.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:01.497 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126142+0000", , 2022-09-04T14:39:01.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:01.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.131673+0000", , 2022-09-04T14:39:01.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9819999999999996e-06 , 2022-09-04T14:39:01.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.188599+0000", , 2022-09-04T14:39:01.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7813999999999997e-05 , 2022-09-04T14:39:01.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:01.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:21.188754+0000", , 2022-09-04T14:39:01.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1012e-05 , 2022-09-04T14:39:01.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:02.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:39:02.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.722763+0000", , 2022-09-04T14:39:02.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:02.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:39:02.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.724126+0000", , 2022-09-04T14:39:02.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.725e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:02.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:39:02.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.737326+0000", , 2022-09-04T14:39:02.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076465750000000001 Ignoring malparsed backtrace: 2022-09-04T14:39:02.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:39:02.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.738323+0000", , 2022-09-04T14:39:02.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083854519999999998 Ignoring malparsed backtrace: 2022-09-04T14:39:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.753392+0000", , 2022-09-04T14:39:02.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4231e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:39:02.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:12.762617+0000", , 2022-09-04T14:39:02.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00086305099999999999 Ignoring malparsed backtrace: 2022-09-04T14:39:02.252 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:39:02.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.830556+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:02.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.860982+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:02.301 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:39:02.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.798464+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:02.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:39:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:02.346 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:02.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:39:02.369 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:39:02.369 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:39:02.369 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:39:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:02.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:02.392 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000147532 , 2022-09-04T14:39:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:02.414 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148901 , 2022-09-04T14:39:02.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:02.443 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:02.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:15.165460+0000", , 2022-09-04T14:39:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.454e-06 , 2022-09-04T14:39:02.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:02.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7264e-05 , 2022-09-04T14:39:02.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:02.947 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:02.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8272999999999999e-05 , 2022-09-04T14:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:02.971 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:02.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682691+0000", , 2022-09-04T14:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:02.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:03.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.156307+0000", , 2022-09-04T14:39:03.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120665 , 2022-09-04T14:39:03.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:03.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.560710+0000", , 2022-09-04T14:39:03.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6932e-05 , 2022-09-04T14:39:03.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:03.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:39:03.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910004+0000", , 2022-09-04T14:39:03.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:03.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:39:03.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:41.634557+0000", , 2022-09-04T14:39:03.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4173999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:03.119 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:03.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:03.120 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:03.144 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:03.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:03.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.087737+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:03.168 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:03.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:03.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.090574+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:03.192 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:39:03.192 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:03.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:03.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:03.217 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:03.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:39:03.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:03.243 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:03.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:39:03.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:03.269 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:03.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:03.293 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:03.293 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:39:03.294 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:03.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false , 2022-09-04T14:39:03.816 INFO:teuthology.orchestra.run.smithi038.stdout:{ , 2022-09-04T14:39:03.817 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, Ignoring malparsed backtrace: 2022-09-04T14:39:03.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:18.783973+0000", , 2022-09-04T14:39:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:39:03.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:03.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833188+0000", , 2022-09-04T14:39:03.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8930000000000005e-06 , 2022-09-04T14:39:03.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.840927+0000", , 2022-09-04T14:39:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067311599999999999 , 2022-09-04T14:39:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:39:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834326+0000", , 2022-09-04T14:39:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:39:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:39:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.777357+0000", , 2022-09-04T14:39:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1764e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:39:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.786750+0000", , 2022-09-04T14:39:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070554800000000003 Ignoring malparsed backtrace: 2022-09-04T14:39:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:39:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.908022+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:39:04.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910185+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:39:04.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.937736+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:04.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:39:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:04.109 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:04.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:39:04.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:04.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:39:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3062999999999997e-05 , 2022-09-04T14:39:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011418820000000001 , 2022-09-04T14:39:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:05.526+0000 7fc80fcb9700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 39177) UID: 0 , 2022-09-04T14:39:05.597 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:39:05.597 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: 2022-09-04T14:39:06.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:06.914 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:06.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:06.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:06.945 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:06.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:06.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:06.969 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:06.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:06.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:06.994 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:06.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:07.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:07.018 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:07.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:39:07.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:07.043 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:07.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:39:07.067 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:159 3.5s0 3:a83f9681:::smithi16339483-3:head [copy-from ver 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:39:07.067 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.858420+0000", , 2022-09-04T14:39:07.067 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 294.03406629400001, Ignoring malparsed backtrace: 2022-09-04T14:39:07.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:07.091 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ , 2022-09-04T14:39:07.092 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:07.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2344000000000001e-05 , 2022-09-04T14:39:07.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:07.116 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:07.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4259999999999992e-06 , 2022-09-04T14:39:07.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:07.140 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:39:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:39:07.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801835+0000", , 2022-09-04T14:39:07.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7821999999999998e-05 , 2022-09-04T14:39:07.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:07.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833134+0000", , 2022-09-04T14:39:07.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030157069999999998 , 2022-09-04T14:39:07.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:07.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.909173+0000", , 2022-09-04T14:39:07.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003982925 , 2022-09-04T14:39:07.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:07.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098949610000000007, , 2022-09-04T14:39:07.267 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:39:07.267 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2022-09-04T14:39:07.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084413999999999999, , 2022-09-04T14:39:07.293 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:39:07.293 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:39:07.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:39:07.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:22.192250+0000", , 2022-09-04T14:39:07.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1777999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:07.815 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:39:07.815 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:39:07.815 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:39:07.848 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:39:07.848 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:39:07.848 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { Ignoring malparsed backtrace: 2022-09-04T14:39:07.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858391+0000", , 2022-09-04T14:39:07.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999986 , 2022-09-04T14:39:07.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:07.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858439+0000", , 2022-09-04T14:39:07.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6209999999999998e-06 , 2022-09-04T14:39:07.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:07.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.802043+0000", , 2022-09-04T14:39:07.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2689999999999999e-06 , 2022-09-04T14:39:07.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:07.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0076027159999999998, , 2022-09-04T14:39:07.956 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:39:07.956 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:39:07.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:39:07.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.841091+0000", , 2022-09-04T14:39:07.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.951e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:08.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:08.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.914176+0000", , 2022-09-04T14:39:08.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1678999999999998e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:08.044 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:08.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:39:08.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949007+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:08.068 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:08.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:08.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949225+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:08.098 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:45.951119+0000", , 2022-09-04T14:39:08.098 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 261.85862011400002, , 2022-09-04T14:39:08.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098030870000000003, Ignoring malparsed backtrace: 2022-09-04T14:39:08.129 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:45.951154+0000", , 2022-09-04T14:39:08.129 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 261.85858504399999, , 2022-09-04T14:39:08.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.009986623, Ignoring malparsed backtrace: 2022-09-04T14:39:08.152 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:58.069344+0000", , 2022-09-04T14:39:08.153 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 249.74039447000001, , 2022-09-04T14:39:08.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010420637, Ignoring malparsed backtrace: 2022-09-04T14:39:08.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:08.177 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:08.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:08.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:08.201 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:08.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:08.224 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.5s3 100/33 ECSubWrite(tid=70, reqid=client.4557.0:918, at_version=100'52, trim_to=0'0, roll_forward_to=97'50))", , 2022-09-04T14:39:08.225 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:35:17.175684+0000", , 2022-09-04T14:39:08.225 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 230.634054612, Ignoring malparsed backtrace: 2022-09-04T14:39:08.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, , 2022-09-04T14:39:08.663 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ , 2022-09-04T14:39:08.663 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:08.699 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:08.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:39:08.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.779855+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:08.724 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:08.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:39:08.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.841110+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:08.753 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:08.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:39:08.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.844107+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:08.782 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:39:08.782 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:39:08.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:39:08.807 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:39:08.808 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:39:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:08.830 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:39:08.852 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:08.853 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:39:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:08.876 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:08.898 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:08.899 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:08.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:08.921 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:39:08.921 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:08.943 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9246000000000003e-05 , 2022-09-04T14:39:08.943 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:08.943 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:08.965 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:09.116548+0000", , 2022-09-04T14:39:08.966 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7519999999999996e-06 , 2022-09-04T14:39:08.966 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:08.989 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173058+0000", , 2022-09-04T14:39:08.989 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000198846 , 2022-09-04T14:39:08.989 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:09.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:39:09.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.247367+0000", , 2022-09-04T14:39:09.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:09.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:39:09.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:21.247411+0000", , 2022-09-04T14:39:09.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:39:09.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:39:09.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.728017+0000", , 2022-09-04T14:39:09.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:09.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:39:09.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.728044+0000", , 2022-09-04T14:39:09.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:09.561 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:09.561 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.754559+0000", , 2022-09-04T14:39:09.561 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1620000000000001e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:09.583 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", , 2022-09-04T14:39:09.583 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.777885+0000", , 2022-09-04T14:39:09.584 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8360000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:09.606 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", , 2022-09-04T14:39:09.606 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778521+0000", , 2022-09-04T14:39:09.607 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1198999999999994e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:09.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", , 2022-09-04T14:39:09.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.789808+0000", , 2022-09-04T14:39:09.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085806739999999999 Ignoring malparsed backtrace: 2022-09-04T14:39:09.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", , 2022-09-04T14:39:09.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.792301+0000", , 2022-09-04T14:39:09.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013443800000000001 Ignoring malparsed backtrace: 2022-09-04T14:39:09.674 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:39:09.674 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:39:09.674 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 95 Ignoring malparsed backtrace: 2022-09-04T14:39:09.697 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:39:09.697 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:39:09.697 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 96 Ignoring malparsed backtrace: 2022-09-04T14:39:09.720 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:39:09.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.821435+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:09.743 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", , 2022-09-04T14:39:09.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.791189+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:09.767 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", , 2022-09-04T14:39:09.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.867825+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:09.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:09.790 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.790 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:09.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:09.814 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.814 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:09.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:09.837 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.837 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:39:09.867 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:09.867 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.867 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:39:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:39:09.919 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} , 2022-09-04T14:39:09.821+0000 7f4570c1c700 0 -- 172.21.15.38:0/1950977310 >> [v2:172.21.15.163:6808/38951,v1:172.21.15.163:6809/38951] conn(0x7f456c218d80 msgr2=0x7f456c2191f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure , 2022-09-04T14:39:09.929 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} Ignoring malparsed backtrace: 2022-09-04T14:39:12.111 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:39:12.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops , 2022-09-04T14:39:12.132+0000 7ffa4ea67700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 38946) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:39:12.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1839999999999999e-06 , 2022-09-04T14:39:12.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:12.725 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:12.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003037676 , 2022-09-04T14:39:12.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:12.749 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833106+0000", , 2022-09-04T14:39:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833944+0000", , 2022-09-04T14:39:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5460000000000003e-06 , 2022-09-04T14:39:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.834671+0000", , 2022-09-04T14:39:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9216999999999996e-05 , 2022-09-04T14:39:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:12.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:39:12.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.778827+0000", , 2022-09-04T14:39:12.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:12.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:12.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.022865+0000", , 2022-09-04T14:39:12.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6020000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.908407+0000", , 2022-09-04T14:39:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4618999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:39:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910014+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916356+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954969+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:39:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:39:13.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:13.014 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:13.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:13.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:13.037 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:13.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:13.060 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:13.061 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:39:13.061 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:13.491 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:39:13.491 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:13.491 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:39:13.514 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:13.514 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:13.514 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:13.536 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:13.536 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:13.536 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:13.559 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:13.559 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ , 2022-09-04T14:39:13.559 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:13.582 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.2351999999999998e-05 , 2022-09-04T14:39:13.582 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:13.582 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:13.604 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.3790000000000003e-06 , 2022-09-04T14:39:13.604 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:13.604 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:13.626 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:39:13.627 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:13.627 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:39:13.649 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816529+0000", , 2022-09-04T14:39:13.650 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.5992999999999998e-05 , 2022-09-04T14:39:13.650 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:13.672 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.686383+0000", , 2022-09-04T14:39:13.672 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.896e-06 , 2022-09-04T14:39:13.673 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:13.695 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.66952835799999999, , 2022-09-04T14:39:13.695 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:39:13.695 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:39:13.717 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:39:13.717 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:51.083722+0000", , 2022-09-04T14:39:13.717 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.105e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:13.740 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:13.740 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.069807+0000", , 2022-09-04T14:39:13.740 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.9273000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 244.35559287000001, , 2022-09-04T14:39:13.762 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0039824329999999996, , 2022-09-04T14:39:13.763 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:39:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:39:13.785 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:21.265193+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:13.808 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:13.808 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:13.808 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192553+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:13.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight , 2022-09-04T14:39:13.831+0000 7faf5177f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38712) UID: 0 , 2022-09-04T14:39:13.855+0000 7f4035433700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 3a=[]:{e=[e,c,b,9],1b=[1b,1a,17,14],23=[23,22,1f,1e],3a=[3a,39,34,33]} for object 3:96c14a12:::smithi16339483-31:8 Ignoring malparsed backtrace: 2022-09-04T14:39:14.340 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00027289000000000002 , 2022-09-04T14:39:14.340 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:39:14.340 INFO:teuthology.orchestra.run.smithi077.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:39:14.368 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.819222+0000", , 2022-09-04T14:39:14.368 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.999999 , 2022-09-04T14:39:14.369 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:14.397 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834523+0000", , 2022-09-04T14:39:14.398 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.1673e-05 , 2022-09-04T14:39:14.398 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:14.430 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.845055+0000", , 2022-09-04T14:39:14.430 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0083501570000000004 , 2022-09-04T14:39:14.430 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:14.460 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:39:14.460 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.842241+0000", , 2022-09-04T14:39:14.460 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967296 Ignoring malparsed backtrace: 2022-09-04T14:39:14.488 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:39:14.489 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843097+0000", , 2022-09-04T14:39:14.489 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 9.4860000000000003e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:14.513 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:39:14.513 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.849421+0000", , 2022-09-04T14:39:14.514 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0052858799999999997 Ignoring malparsed backtrace: 2022-09-04T14:39:14.538 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:14.538 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", , 2022-09-04T14:39:14.538 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684517+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:14.564 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:14.564 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", , 2022-09-04T14:39:14.564 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:35.277437+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:14.590 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:14.590 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", , 2022-09-04T14:39:14.590 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973030+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:14.616 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:14.616 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:14.616 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:39:14.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:14.641 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:14.642 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:39:14.666 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:14.666 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:14.667 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:39:14.691 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:39:14.691 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:14.691 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:14.716 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 5.4320999999999997e-05 , 2022-09-04T14:39:14.716 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:14.716 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:15.139 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:15.139 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:39:15.140 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.783312+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:15.164 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:15.164 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:15.164 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.785248+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:15.187 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:15.187 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:15.188 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:39:15.276 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:15.276 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:15.276 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:15.301 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:15.301 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:15.301 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:39:15.326 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:39:15.326 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:15.327 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:15.351 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.2610000000000004e-06 , 2022-09-04T14:39:15.351 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:15.351 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:15.376 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.2515000000000002e-05 , 2022-09-04T14:39:15.376 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:15.376 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:15.400 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973128+0000", , 2022-09-04T14:39:15.401 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:39:15.401 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:15.425 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.070679+0000", , 2022-09-04T14:39:15.425 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.075e-05 , 2022-09-04T14:39:15.425 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:15.450 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:59.067302+0000", , 2022-09-04T14:39:15.450 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 9.3108000000000004e-05 , 2022-09-04T14:39:15.450 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:15.475 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:39:15.475 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.081257+0000", , 2022-09-04T14:39:15.475 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:15.499 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:15.499 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:03.083959+0000", , 2022-09-04T14:39:15.500 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4710000000000004e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:15.524 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:15.524 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:09.119460+0000", , 2022-09-04T14:39:15.524 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.0303999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:15.549 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:15.549 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "initiated", , 2022-09-04T14:39:15.549 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194319+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:16.162 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:39:16.163 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:39:16.170+0000 7efdf07f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 38715) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:39:18.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops , 2022-09-04T14:39:18.088+0000 7faf5177f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 38712) UID: 0 , 2022-09-04T14:39:18.163 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument Ignoring malparsed backtrace: 2022-09-04T14:39:19.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:19.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:19.592 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:19.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9209999999999992e-06 , 2022-09-04T14:39:19.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:19.623 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:19.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6297e-05 , 2022-09-04T14:39:19.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:19.656 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:19.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682622+0000", , 2022-09-04T14:39:19.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4299999999999995e-06 , 2022-09-04T14:39:19.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:19.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.741685+0000", , 2022-09-04T14:39:19.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1010000000000002e-05 , 2022-09-04T14:39:19.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:19.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:39:19.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910400+0000", , 2022-09-04T14:39:19.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:19.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:39:19.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.962849+0000", , 2022-09-04T14:39:19.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1389999999999997e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:19.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:39:19.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.965767+0000", , 2022-09-04T14:39:19.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029028639999999998 Ignoring malparsed backtrace: 2022-09-04T14:39:19.804 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:19.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:19.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:50.983637+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:19.828 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:19.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:19.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.121606+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:39:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:19.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:19.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:19.874 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:19.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:39:19.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:19.896 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:19.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:39:19.918 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:19.919 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:39:19.919 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:19.947 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:19.947 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:39:19.948 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:20.112+0000 7f8152796700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset e=[]:{e=[e,c,b,9]} for object 3:0dcdd893:::smithi16339483-22:8 , 2022-09-04T14:39:20.145 INFO:teuthology.orchestra.run.smithi038.stdout:{ , 2022-09-04T14:39:20.146 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], Ignoring malparsed backtrace: 2022-09-04T14:39:20.461 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:33 3.bs0 3:dbd93fd6:::smithi16339483-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", , 2022-09-04T14:39:20.461 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:12.724116+0000", , 2022-09-04T14:39:20.461 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 307.70282368400001, Ignoring malparsed backtrace: 2022-09-04T14:39:20.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:20.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:20.489 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:20.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:20.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:20.513 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:20.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8459999999999999e-06 , 2022-09-04T14:39:20.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:20.537 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:20.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.313e-06 , 2022-09-04T14:39:20.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:20.561 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:20.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9201e-05 , 2022-09-04T14:39:20.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:20.584 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:20.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0903999999999996e-05 , 2022-09-04T14:39:20.607 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:20.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:39:20.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.780112+0000", , 2022-09-04T14:39:20.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4380000000000003e-06 , 2022-09-04T14:39:20.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:20.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.909356+0000", , 2022-09-04T14:39:20.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4094000000000002e-05 , 2022-09-04T14:39:20.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:20.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", , 2022-09-04T14:39:20.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916279+0000", , 2022-09-04T14:39:20.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:20.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:39:20.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.976998+0000", , 2022-09-04T14:39:20.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4039000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:59.075202+0000", , 2022-09-04T14:39:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169333 Ignoring malparsed backtrace: 2022-09-04T14:39:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:39:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.101200+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:03.101233+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { , 2022-09-04T14:39:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4557", , 2022-09-04T14:39:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:39:21.242 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:39:21.242 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:21.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:39:21.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:21.277 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:21.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:21.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:21.303 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:21.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ , 2022-09-04T14:39:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.40320693000000002 , 2022-09-04T14:39:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:21.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0587e-05 , 2022-09-04T14:39:21.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:21.380 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { , 2022-09-04T14:39:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:39:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.963666+0000", , 2022-09-04T14:39:21.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001010831 , 2022-09-04T14:39:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.078290+0000", , 2022-09-04T14:39:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4528999999999999e-05 , 2022-09-04T14:39:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:21.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:39:21.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:58.090354+0000", , 2022-09-04T14:39:21.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:21.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:39:21.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126454+0000", , 2022-09-04T14:39:21.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107756 Ignoring malparsed backtrace: 2022-09-04T14:39:21.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:21.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.130038+0000", , 2022-09-04T14:39:21.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2265999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:21.554 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:21.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:39:21.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.133696+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:21.579 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:21.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:21.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:10.127233+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:21.605 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:21.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", , 2022-09-04T14:39:21.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:16.171854+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:21.630 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:21.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:21.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:16.172953+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3348000000000001e-05 , 2022-09-04T14:39:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:22.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025171820000000002 , 2022-09-04T14:39:22.157 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:22.158 INFO:teuthology.orchestra.run.smithi038.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:39:22.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833106+0000", , 2022-09-04T14:39:22.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:39:22.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:22.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.833998+0000", , 2022-09-04T14:39:22.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7340000000000003e-06 , 2022-09-04T14:39:22.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:22.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:23.841385+0000", , 2022-09-04T14:39:22.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067012039999999997 , 2022-09-04T14:39:22.237 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:39:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:27.778827+0000", , 2022-09-04T14:39:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:39:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:39:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:35.022943+0000", , 2022-09-04T14:39:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3577000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:39:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.921652+0000", , 2022-09-04T14:39:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013170645 Ignoring malparsed backtrace: 2022-09-04T14:39:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", , 2022-09-04T14:39:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910014+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", , 2022-09-04T14:39:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.916415+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", , 2022-09-04T14:39:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.961162+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", Ignoring malparsed backtrace: 2022-09-04T14:39:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_started", Ignoring malparsed backtrace: 2022-09-04T14:39:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:39:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:23.015+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:39:23.066 INFO:teuthology.orchestra.run.smithi077.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:39:23.066 INFO:teuthology.orchestra.run.smithi077.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. Ignoring malparsed backtrace: osd.9: osd_enable_op_tracker = 'true' , osd.10: osd_enable_op_tracker = 'true' , osd.11: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", Ignoring malparsed backtrace: 2022-09-04T14:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", , 2022-09-04T14:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999986 , 2022-09-04T14:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:25.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1809999999999998e-06 , 2022-09-04T14:39:25.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:25.191 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:25.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4130000000000003e-06 , 2022-09-04T14:39:25.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:25.216 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:25.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0072074699999999997 , 2022-09-04T14:39:25.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:25.249 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:25.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.11185637900000001 , 2022-09-04T14:39:25.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:25.278 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:25.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783674+0000", , 2022-09-04T14:39:25.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:39:25.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:25.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.784617+0000", , 2022-09-04T14:39:25.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6599999999999998e-06 , 2022-09-04T14:39:25.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:25.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.785148+0000", , 2022-09-04T14:39:25.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6949000000000001e-05 , 2022-09-04T14:39:25.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:25.379 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:39:25.379 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 300 , 2022-09-04T14:39:25.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:25.404 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:39:25.404 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 483 , 2022-09-04T14:39:25.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:25.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", , 2022-09-04T14:39:25.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:56.032036+0000", , 2022-09-04T14:39:25.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:25.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", , 2022-09-04T14:39:25.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:17.174904+0000", , 2022-09-04T14:39:25.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103144 Ignoring malparsed backtrace: 2022-09-04T14:39:25.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:25.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:21.268867+0000", , 2022-09-04T14:39:25.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024077450000000002 Ignoring malparsed backtrace: 2022-09-04T14:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout:{ , 2022-09-04T14:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], , 2022-09-04T14:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:26.012 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:26.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:26.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.753359+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:26.042 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:26.043 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:26.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:312 3.cs0 3:38890410:::smithi16339483-19:head [append 0~0] snapc c=[c,b,9,8,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", , 2022-09-04T14:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:23.841089+0000", , 2022-09-04T14:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 302.14747684600002, Ignoring malparsed backtrace: 2022-09-04T14:39:26.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:39:26.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:26.185 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:26.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3142999999999999e-05 , 2022-09-04T14:39:26.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:26.215 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:26.241 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949068+0000", , 2022-09-04T14:39:26.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:39:26.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:26.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949251+0000", , 2022-09-04T14:39:26.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000146267 , 2022-09-04T14:39:26.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.954009+0000", , 2022-09-04T14:39:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7460000000000003e-06 , 2022-09-04T14:39:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:26.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:45.954718+0000", , 2022-09-04T14:39:26.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2200000000000003e-06 , 2022-09-04T14:39:26.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:26.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:26.353 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:26.353 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.1s1 81/52 ECSubWrite(tid=203, reqid=client.4557.0:689, at_version=81'36, trim_to=0'0, roll_forward_to=66'34))", Ignoring malparsed backtrace: 2022-09-04T14:39:26.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:39:26.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:58.071680+0000", , 2022-09-04T14:39:26.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0654e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:26.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:26.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:09.119489+0000", , 2022-09-04T14:39:26.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018941300000000001 Ignoring malparsed backtrace: 2022-09-04T14:39:26.434 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:26.435 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:26.435 INFO:teuthology.orchestra.run.smithi155.stdout: ] Ignoring malparsed backtrace: 2022-09-04T14:39:26.872 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6672000000000003e-05 , 2022-09-04T14:39:26.873 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:26.873 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:26.909 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:26.909 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:308 3.2s1 3:4f5d92c8:::smithi16339483-43:head [setxattr _header (56) in=7b] snapc c=[c,b,9,8,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", , 2022-09-04T14:39:26.910 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:23.841104+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:26.935 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 309 , 2022-09-04T14:39:26.935 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:26.936 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:39:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.843238+0000", , 2022-09-04T14:39:26.960 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5185000000000002e-05 , 2022-09-04T14:39:26.961 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.681074+0000", , 2022-09-04T14:39:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1880000000000002e-06 , 2022-09-04T14:39:26.985 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:27.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683681+0000", , 2022-09-04T14:39:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000101047 , 2022-09-04T14:39:27.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.696200+0000", , 2022-09-04T14:39:27.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6873000000000002e-05 , 2022-09-04T14:39:27.039 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:27.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011618695, , 2022-09-04T14:39:27.064 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:39:27.064 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", Ignoring malparsed backtrace: 2022-09-04T14:39:27.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", , 2022-09-04T14:39:27.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.069393+0000", , 2022-09-04T14:39:27.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5677000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting_for_map not empty", , 2022-09-04T14:39:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.046883+0000", , 2022-09-04T14:39:27.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8390000000000003e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:27.137 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:27.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:39:27.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.051786+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:27.162 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:09.116537+0000", , 2022-09-04T14:39:27.162 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 257.73355048100001, , 2022-09-04T14:39:27.163 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.46953788800000001, Ignoring malparsed backtrace: 2022-09-04T14:39:27.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:27.187 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:27.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:27.211 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:27.212 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:27.212 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:27.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:27.238 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:27.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", Ignoring malparsed backtrace: 2022-09-04T14:39:27.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops , 2022-09-04T14:39:27.464+0000 7ffa4ea67700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 38946) UID: 0 , 2022-09-04T14:39:27.478 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} Ignoring malparsed backtrace: 2022-09-04T14:39:28.982 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:39:28.983 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:39:28.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight Ignoring malparsed backtrace: 2022-09-04T14:39:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:18.783233+0000", , 2022-09-04T14:39:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 311.72581230499998, , 2022-09-04T14:39:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064063829999999999, Ignoring malparsed backtrace: 2022-09-04T14:39:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:30.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:30.567 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:30.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:30.608 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s4 47/33 ECSubWrite(tid=21, reqid=client.4557.0:300, at_version=47'15, trim_to=0'0, roll_forward_to=37'11))", , 2022-09-04T14:39:30.608 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:23.833892+0000", , 2022-09-04T14:39:30.609 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 306.67515301100002, Ignoring malparsed backtrace: 2022-09-04T14:39:30.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7445999999999999e-05 , 2022-09-04T14:39:30.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:30.650 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:30.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012869300000000001 , 2022-09-04T14:39:30.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:30.683 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:30.717 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:30.717 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.es6 58/33 ECSubWrite(tid=19, reqid=osd.11.0:19, at_version=58'14, trim_to=0'0, roll_forward_to=54'12))", , 2022-09-04T14:39:30.718 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:35.022799+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:30.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.908050+0000", , 2022-09-04T14:39:30.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7835999999999999e-05 , 2022-09-04T14:39:30.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910303+0000", , 2022-09-04T14:39:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117623 , 2022-09-04T14:39:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDECSubOpWrite(3.5s4 63/33 ECSubWrite(tid=40, reqid=client.4557.0:483, at_version=0'0, trim_to=0'0, roll_forward_to=63'27))", Ignoring malparsed backtrace: 2022-09-04T14:39:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.954348+0000", , 2022-09-04T14:39:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4910999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.955055+0000", , 2022-09-04T14:39:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5217999999999995e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", , 2022-09-04T14:39:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.126563+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:30.942 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:30.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:30.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:35:09.130236+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:30.966 INFO:teuthology.orchestra.run.smithi038.stdout: ] , 2022-09-04T14:39:30.966 INFO:teuthology.orchestra.run.smithi038.stdout: } , 2022-09-04T14:39:30.966 INFO:teuthology.orchestra.run.smithi038.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:31.400 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.781888+0000", , 2022-09-04T14:39:31.400 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.1753000000000001e-05 , 2022-09-04T14:39:31.400 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:31.424 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:39:31.424 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.783649+0000", , 2022-09-04T14:39:31.424 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:31.448 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:39:31.448 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.784659+0000", , 2022-09-04T14:39:31.448 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.0491999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:31.475 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:31.475 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.785559+0000", , 2022-09-04T14:39:31.475 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.4518999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:31.502 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:31.502 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:39:31.502 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.789213+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:31.525 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:31.525 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:39:31.526 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.790678+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:31.549 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:31.549 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:31.549 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:18.807882+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:31.572 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:31.572 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:31.573 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:39:31.597 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:31.597 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:31.598 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:31.621 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:31.622 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:31.622 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:39:31.644 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:39:31.644 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:31.645 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:31.669 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.9020000000000002e-06 , 2022-09-04T14:39:31.669 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:31.669 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:31.697 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.00019496100000000001 , 2022-09-04T14:39:31.698 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:31.698 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:31.727 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.192188+0000", , 2022-09-04T14:39:31.727 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:39:31.728 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:31.753 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:35:22.194368+0000", , 2022-09-04T14:39:31.753 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.3180000000000007e-06 , 2022-09-04T14:39:31.753 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:32.252 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.035869856999999998 , 2022-09-04T14:39:32.253 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:32.253 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:32.276 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.6820000000000003e-06 , 2022-09-04T14:39:32.277 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:32.277 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:32.301 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:39:32.301 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:32.301 INFO:teuthology.orchestra.run.smithi077.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:39:32.325 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.835507+0000", , 2022-09-04T14:39:32.325 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.001094898 , 2022-09-04T14:39:32.325 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:32.349 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:24.598551+0000", , 2022-09-04T14:39:32.349 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.0975000000000001e-05 , 2022-09-04T14:39:32.349 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:32.373 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0064595980000000004, , 2022-09-04T14:39:32.373 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { , 2022-09-04T14:39:32.373 INFO:teuthology.orchestra.run.smithi077.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:39:32.396 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:39:32.397 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843624+0000", , 2022-09-04T14:39:32.397 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.5616999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:32.421 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:32.421 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684810+0000", , 2022-09-04T14:39:32.421 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 6.4609999999999996e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:32.444 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 297.21970047899998, , 2022-09-04T14:39:32.445 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.25616860400000002, , 2022-09-04T14:39:32.445 INFO:teuthology.orchestra.run.smithi077.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:39:32.468 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:32.468 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", , 2022-09-04T14:39:32.469 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.968564+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:32.502 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:32.503 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:32.503 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.972473+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:32.526 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:46.966547+0000", , 2022-09-04T14:39:32.526 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 285.275186195, , 2022-09-04T14:39:32.527 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.0046854039999999998, Ignoring malparsed backtrace: 2022-09-04T14:39:32.550 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:32.550 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:32.550 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:32.573 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:32.573 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:32.574 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:32.597 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.0s4 86/33 ECSubWrite(tid=128, reqid=client.4557.0:793, at_version=0'0, trim_to=0'0, roll_forward_to=86'71))", , 2022-09-04T14:39:32.598 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:35:03.105466+0000", , 2022-09-04T14:39:32.598 INFO:teuthology.orchestra.run.smithi077.stdout: "age": 269.13626772700002, Ignoring malparsed backtrace: 2022-09-04T14:39:32.898 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} , 2022-09-04T14:39:32.921 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} , 2022-09-04T14:39:32.938+0000 7fce19959700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 38719) UID: 0 Ignoring malparsed backtrace: 2022-09-04T14:39:34.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight , 2022-09-04T14:39:34.939+0000 7f9335f09700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 38716) UID: 0 , 2022-09-04T14:39:34.995+0000 7f793eff5700 -1 log_channel(cluster) log [ERR] : No clone_snaps in snapset 43=[]:{17=[17,14,12,e,c],27=[27,25,23,22,1f],43=[42]} for object 3:a9d45bed:::smithi16339483-27:2 Ignoring malparsed backtrace: 2022-09-04T14:39:35.407 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:39:35.407 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:39:35.407 INFO:teuthology.orchestra.run.smithi163.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:35.430 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:39:35.430 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:39:35.431 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:39:35.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:39:35.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:35.457 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:35.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1739999999999997e-06 , 2022-09-04T14:39:35.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:35.481 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:35.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7080999999999999e-05 , 2022-09-04T14:39:35.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:35.505 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:35.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012313925 , 2022-09-04T14:39:35.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:35.528 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:35.551 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:35.551 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.4557.0:635 3.2s1 3:402e69e6:::smithi16339483-26:head [copy-from ver 48] snapc 23=[23,22,1f,1e,1b,14] ondisk+write+known_if_redirected+supports_pool_eio e74)", , 2022-09-04T14:39:35.551 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:51.073653+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:35.574 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 636 , 2022-09-04T14:39:35.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:35.575 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:39:35.597 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:58.069441+0000", , 2022-09-04T14:39:35.598 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7750000000000002e-05 , 2022-09-04T14:39:35.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:35.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.048774+0000", , 2022-09-04T14:39:35.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018909280000000001 , 2022-09-04T14:39:35.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:35.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:39:35.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:59.051786+0000", , 2022-09-04T14:39:35.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:35.667 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", , 2022-09-04T14:39:35.667 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:39:35.668 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 855 Ignoring malparsed backtrace: 2022-09-04T14:39:35.694 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:35.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:39:35.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.171805+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:35.717 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:35.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:35.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173555+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:35.740 INFO:teuthology.orchestra.run.smithi163.stdout: ] , 2022-09-04T14:39:35.740 INFO:teuthology.orchestra.run.smithi163.stdout: } , 2022-09-04T14:39:35.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: osd.2: osd_enable_op_tracker = 'true' , osd.3: osd_enable_op_tracker = 'true' , osd.4: osd_enable_op_tracker = 'true' Ignoring malparsed backtrace: 2022-09-04T14:39:36.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", , 2022-09-04T14:39:36.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.733956+0000", , 2022-09-04T14:39:36.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00056110299999999997 Ignoring malparsed backtrace: 2022-09-04T14:39:36.211 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:39:36.212 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.742180+0000", , 2022-09-04T14:39:36.212 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0079690790000000004 Ignoring malparsed backtrace: 2022-09-04T14:39:36.237 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 323.39142833300002, , 2022-09-04T14:39:36.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013955565, , 2022-09-04T14:39:36.237 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:39:36.263 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 323.39138634, , 2022-09-04T14:39:36.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016237340999999999, , 2022-09-04T14:39:36.263 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:39:36.289 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:36.289 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", , 2022-09-04T14:39:36.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778219+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:36.314 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:36.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:39:36.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778556+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:36.340 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:36.340 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:39:36.340 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.778601+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:36.364 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:36.364 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:39:36.365 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.791489+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:36.390 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:36.390 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:39:36.391 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.792114+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:36.416 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:36.416 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", , 2022-09-04T14:39:36.417 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.833527+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:36.442 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:18.787775+0000", , 2022-09-04T14:39:36.442 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 317.37862695600001, , 2022-09-04T14:39:36.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.080833129000000004, Ignoring malparsed backtrace: 2022-09-04T14:39:36.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:36.467 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:36.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:36.491 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-09-04T14:39:36.491 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { , 2022-09-04T14:39:36.492 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.4557", Ignoring malparsed backtrace: 2022-09-04T14:39:36.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:39:36.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:36.524 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:36.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 , 2022-09-04T14:39:36.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:36.547 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:36.569 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 , 2022-09-04T14:39:36.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:36.570 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:36.593 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2761e-05 , 2022-09-04T14:39:36.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:36.594 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:37.014 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:37.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776521+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:37.037 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:37.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:12.776564+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:37.061 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:37.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:23.833242+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:37.085 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:39:37.085 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:39:37.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:37.108 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:37.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:37.131 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:37.154 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:37.155 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:39:37.155 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:37.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6502000000000001e-05 , 2022-09-04T14:39:37.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:37.178 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:37.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8776999999999999e-05 , 2022-09-04T14:39:37.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:37.203 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:37.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0055604799999999996 , 2022-09-04T14:39:37.226 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:37.226 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:37.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.170924+0000", , 2022-09-04T14:39:37.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 , 2022-09-04T14:39:37.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:37.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173007+0000", , 2022-09-04T14:39:37.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1848999999999998e-05 , 2022-09-04T14:39:37.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:37.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.173538+0000", , 2022-09-04T14:39:37.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2520999999999999e-05 , 2022-09-04T14:39:37.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:37.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", , 2022-09-04T14:39:37.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.175739+0000", , 2022-09-04T14:39:37.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:37.342 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", , 2022-09-04T14:39:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:22.192364+0000", , 2022-09-04T14:39:37.343 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1828000000000004e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:37.466+0000 7faa4b2eb700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 38943) UID: 0 , 2022-09-04T14:39:37.534 INFO:teuthology.orchestra.run.smithi163.stdout:{ , 2022-09-04T14:39:37.534 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], Ignoring malparsed backtrace: 2022-09-04T14:39:37.870 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078015669999999997, , 2022-09-04T14:39:37.870 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:39:37.870 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2022-09-04T14:39:37.893 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:39:37.893 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:19.063958+0000", , 2022-09-04T14:39:37.894 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.26081081699999997 Ignoring malparsed backtrace: 2022-09-04T14:39:37.916 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:37.917 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.809700+0000", , 2022-09-04T14:39:37.917 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0129999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:37.939 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 310.061293939, , 2022-09-04T14:39:37.939 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058106429999999999, , 2022-09-04T14:39:37.939 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:39:37.962 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.962 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", , 2022-09-04T14:39:37.962 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683466+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:37.985 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:37.985 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:37.985 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.684331+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:38.007 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:30.732801+0000", , 2022-09-04T14:39:38.007 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 307.11110204300002, , 2022-09-04T14:39:38.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0066745110000000002, Ignoring malparsed backtrace: 2022-09-04T14:39:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:38.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:38.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.4s1 69/52 ECSubWrite(tid=36, reqid=client.4557.0:590, at_version=0'0, trim_to=0'0, roll_forward_to=69'18))", , 2022-09-04T14:39:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:34:45.967225+0000", , 2022-09-04T14:39:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 291.876678717, Ignoring malparsed backtrace: 2022-09-04T14:39:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025593429999999999 , 2022-09-04T14:39:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:38.097 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:38.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023814800000000001 , 2022-09-04T14:39:38.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:38.120 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:38.142 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:38.142 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(3.fs2 99/52 ECSubWrite(tid=119, reqid=client.4557.0:916, at_version=99'71, trim_to=0'0, roll_forward_to=89'67))", , 2022-09-04T14:39:38.142 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2022-09-04T14:35:16.172695+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:38.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:16.175481+0000", , 2022-09-04T14:39:38.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1101e-05 , 2022-09-04T14:39:38.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:38.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.272224+0000", , 2022-09-04T14:39:38.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031848899999999998 , 2022-09-04T14:39:38.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:38.813 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument , 2022-09-04T14:39:38.813 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2022-09-04T14:39:38.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:39:40.865 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:39:40.865 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:40.865 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:40.897 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 3.5865999999999998e-05 , 2022-09-04T14:39:40.897 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:40.898 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:40.922 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.6765e-05 , 2022-09-04T14:39:40.922 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:40.922 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:40.949 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.834389+0000", , 2022-09-04T14:39:40.949 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 , 2022-09-04T14:39:40.950 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:40.978 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:24.598439+0000", , 2022-09-04T14:39:40.979 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.8315999999999999e-05 , 2022-09-04T14:39:40.979 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:41.002 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.844031+0000", , 2022-09-04T14:39:41.003 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.4280000000000001e-05 , 2022-09-04T14:39:41.003 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:41.028 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:39:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.843585+0000", , 2022-09-04T14:39:41.029 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:41.056 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:39:41.056 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684729+0000", , 2022-09-04T14:39:41.056 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000163079 Ignoring malparsed backtrace: 2022-09-04T14:39:41.084 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:41.084 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:29.684935+0000", , 2022-09-04T14:39:41.085 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 2.1464e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:41.112 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.112 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", , 2022-09-04T14:39:41.112 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.968393+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:41.142 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.142 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", , 2022-09-04T14:39:41.143 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.971476+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:41.171 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.171 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", , 2022-09-04T14:39:41.171 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973422+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:41.194 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:41.194 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.194 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:39:41.216 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:41.216 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.216 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "reached_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:41.239 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:41.239 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.239 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "sub_op_committed", Ignoring malparsed backtrace: 2022-09-04T14:39:41.264 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} , 2022-09-04T14:39:41.265 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} , 2022-09-04T14:39:41.047+0000 7f7fbe210700 0 -- 172.21.15.38:0/1391555665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7fb806ca50 msgr2=0x7f7fb8156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure Ignoring malparsed backtrace: 2022-09-04T14:39:41.740 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 7.6657999999999997e-05 , 2022-09-04T14:39:41.740 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:41.740 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:41.767 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.768 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.bs6 42/33 ECSubWrite(tid=60, reqid=client.4557.0:197, at_version=0'0, trim_to=0'0, roll_forward_to=42'11))", , 2022-09-04T14:39:41.768 INFO:teuthology.orchestra.run.smithi077.stdout: "initiated_at": "2022-09-04T14:34:18.788242+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:41.793 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.815526+0000", , 2022-09-04T14:39:41.793 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 1.9049e-05 , 2022-09-04T14:39:41.794 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:41.816 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.816963+0000", , 2022-09-04T14:39:41.816 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 0.000102017 , 2022-09-04T14:39:41.816 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:41.839 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:41.839 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.839 INFO:teuthology.orchestra.run.smithi077.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/33 ECSubWrite(tid=82, reqid=client.4557.0:292, at_version=47'27, trim_to=0'0, roll_forward_to=47'25))", Ignoring malparsed backtrace: 2022-09-04T14:39:41.861 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:39:41.861 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.832847+0000", , 2022-09-04T14:39:41.861 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4.1347000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:41.883 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:41.884 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:23.833374+0000", , 2022-09-04T14:39:41.884 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 8.8985000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:41.906 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:39:41.906 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:41.906 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:41.929 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.929 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", , 2022-09-04T14:39:41.929 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:39.973140+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:41.951 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.951 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", , 2022-09-04T14:39:41.952 INFO:teuthology.orchestra.run.smithi077.stdout: "time": "2022-09-04T14:34:58.071009+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:39:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:39:41.974 INFO:teuthology.orchestra.run.smithi077.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:41.996 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:41.997 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:41.997 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:39:42.019 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:42.019 INFO:teuthology.orchestra.run.smithi077.stdout: { , 2022-09-04T14:39:42.020 INFO:teuthology.orchestra.run.smithi077.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:39:42.042 INFO:teuthology.orchestra.run.smithi077.stdout: } , 2022-09-04T14:39:42.042 INFO:teuthology.orchestra.run.smithi077.stdout: ] , 2022-09-04T14:39:42.043 INFO:teuthology.orchestra.run.smithi077.stdout: } Ignoring malparsed backtrace: 2022-09-04T14:39:42.066 INFO:teuthology.orchestra.run.smithi077.stdout: "duration": 4294967295.999999 , 2022-09-04T14:39:42.066 INFO:teuthology.orchestra.run.smithi077.stdout: }, , 2022-09-04T14:39:42.066 INFO:teuthology.orchestra.run.smithi077.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:42.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:39:42.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.826412+0000", , 2022-09-04T14:39:42.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:42.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", , 2022-09-04T14:39:42.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828860+0000", , 2022-09-04T14:39:42.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00037475099999999999 Ignoring malparsed backtrace: 2022-09-04T14:39:42.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:42.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.828921+0000", , 2022-09-04T14:39:42.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.915e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:42.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", , 2022-09-04T14:39:42.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:21.830545+0000", , 2022-09-04T14:39:42.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9315999999999999e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:42.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:42.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.819438+0000", , 2022-09-04T14:39:42.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0977000000000003e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:42.672 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", , 2022-09-04T14:39:42.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.829701+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:42.695 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:42.695 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:42.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:42.719 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:42.719 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:42.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:42.742 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:42.742 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:42.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:42.766 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:42.766 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:42.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:42.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.790 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:39:42.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.813 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:39:42.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.837 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", Ignoring malparsed backtrace: 2022-09-04T14:39:42.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.860 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:42.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.884 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:42.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.907 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:42.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.930 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:42.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:42.954 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:42.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 , 2022-09-04T14:39:43.146 INFO:teuthology.orchestra.run.smithi155.stdout:} , 2022-09-04T14:39:43.155 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:39:43.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.019991169999999999, , 2022-09-04T14:39:43.454 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { , 2022-09-04T14:39:43.454 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", Ignoring malparsed backtrace: 2022-09-04T14:39:43.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:39:43.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.722488+0000", , 2022-09-04T14:39:43.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:39:43.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:43.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.750513+0000", , 2022-09-04T14:39:43.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099126100000000001 Ignoring malparsed backtrace: 2022-09-04T14:39:43.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:43.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.759663+0000", , 2022-09-04T14:39:43.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4338000000000002e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:43.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:43.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.785356+0000", , 2022-09-04T14:39:43.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.024342609000000001 Ignoring malparsed backtrace: 2022-09-04T14:39:43.573 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:43.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:39:43.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.776540+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:43.596 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:43.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:43.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.800623+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:43.621 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:43.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:43.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.845970+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:43.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.846614+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:43.668 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:43.668 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:43.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:43.692 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { , 2022-09-04T14:39:43.692 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:39:43.693 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", Ignoring malparsed backtrace: 2022-09-04T14:39:43.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:43.716 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:43.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:39:43.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:43.740 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:43.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:43.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:39:43.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:43.764 INFO:teuthology.orchestra.run.smithi155.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:43.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:35:15.158583+0000", , 2022-09-04T14:39:43.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:39:43.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.722250+0000", , 2022-09-04T14:39:44.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4669999999999998e-06 , 2022-09-04T14:39:44.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.722273+0000", , 2022-09-04T14:39:44.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.836e-06 , 2022-09-04T14:39:44.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.767089+0000", , 2022-09-04T14:39:44.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012396 , 2022-09-04T14:39:44.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.767183+0000", , 2022-09-04T14:39:44.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4430999999999993e-05 , 2022-09-04T14:39:44.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.790246+0000", , 2022-09-04T14:39:44.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6967999999999998e-05 , 2022-09-04T14:39:44.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.977772+0000", , 2022-09-04T14:39:44.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9935999999999998e-05 , 2022-09-04T14:39:44.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:44.417 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:44.417 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:159 3.5s0 3:a83f9681:::smithi16339483-3:head [copy-from ver 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", Ignoring malparsed backtrace: 2022-09-04T14:39:44.447 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:39:44.447 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 160 , 2022-09-04T14:39:44.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:44.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:39:44.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783143+0000", , 2022-09-04T14:39:44.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2344000000000001e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:44.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:44.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.783915+0000", , 2022-09-04T14:39:44.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4259999999999992e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:44.522 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 325.41494849399999, , 2022-09-04T14:39:44.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0097379630000000005, , 2022-09-04T14:39:44.523 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { Ignoring malparsed backtrace: 2022-09-04T14:39:44.546 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:44.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", , 2022-09-04T14:39:44.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.801835+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:44.571 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:44.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:44.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.833134+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:44.605 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:44.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", , 2022-09-04T14:39:44.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:39.909173+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:44.628 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:35:17.173569+0000", , 2022-09-04T14:39:44.629 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 267.02617612300003, , 2022-09-04T14:39:44.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0098949610000000007, Ignoring malparsed backtrace: 2022-09-04T14:39:44.652 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:35:21.263683+0000", , 2022-09-04T14:39:44.652 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 262.93606186900001, , 2022-09-04T14:39:44.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0084413999999999999, Ignoring malparsed backtrace: 2022-09-04T14:39:44.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:44.676 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:44.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:45.092 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:45.092 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:56 3.cs0 3:38890410:::smithi16339483-19:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:39:45.093 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.749805+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:45.118 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:45.118 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4557.0:57 3.cs0 3:38890410:::smithi16339483-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", , 2022-09-04T14:39:45.119 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2022-09-04T14:34:12.749826+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:45.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858392+0000", , 2022-09-04T14:39:45.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 , 2022-09-04T14:39:45.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:45.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:12.858433+0000", , 2022-09-04T14:39:45.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3980000000000002e-06 , 2022-09-04T14:39:45.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:45.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:18.802034+0000", , 2022-09-04T14:39:45.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010572400000000001 , 2022-09-04T14:39:45.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:45.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:45.212 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:45.212 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDECSubOpWrite(3.9s4 47/33 ECSubWrite(tid=53, reqid=client.4557.0:283, at_version=47'36, trim_to=0'0, roll_forward_to=45'34))", Ignoring malparsed backtrace: 2022-09-04T14:39:45.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", , 2022-09-04T14:39:45.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.841089+0000", , 2022-09-04T14:39:45.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4294967295.9999995 Ignoring malparsed backtrace: 2022-09-04T14:39:45.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:45.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:23.914125+0000", , 2022-09-04T14:39:45.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0870000000000007e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:45.291 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4557", , 2022-09-04T14:39:45.291 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "172.21.15.163:0/2539419878", , 2022-09-04T14:39:45.292 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 570 Ignoring malparsed backtrace: 2022-09-04T14:39:45.320 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:45.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:45.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2022-09-04T14:34:44.949085+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:45.350 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:45.350 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:45.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:45.384 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:45.384 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:45.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:45.408 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:45.408 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:45.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:45.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:45.432 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:45.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", Ignoring malparsed backtrace: 2022-09-04T14:39:45.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, , 2022-09-04T14:39:45.455 INFO:teuthology.orchestra.run.smithi155.stdout: { , 2022-09-04T14:39:45.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_queued_for_journal_write", Ignoring malparsed backtrace: 2022-09-04T14:39:45.479 INFO:teuthology.orchestra.run.smithi155.stdout: } , 2022-09-04T14:39:45.479 INFO:teuthology.orchestra.run.smithi155.stdout: ] , 2022-09-04T14:39:45.479 INFO:teuthology.orchestra.run.smithi155.stdout: } Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:39:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:39:46.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:39:47.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.800785+0000", , 2022-09-04T14:39:47.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7362999999999996e-05 , 2022-09-04T14:39:47.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:47.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:39:47.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.803132+0000", , 2022-09-04T14:39:47.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:47.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:47.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:18.809473+0000", , 2022-09-04T14:39:47.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3170000000000002e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:47.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:47.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:27.780559+0000", , 2022-09-04T14:39:47.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7598999999999997e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:47.151 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:47.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", , 2022-09-04T14:39:47.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683297+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:47.174 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:47.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:47.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:29.683512+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:47.197 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:47.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", , 2022-09-04T14:39:47.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:34:30.697349+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:47.220 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:39:47.220 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:47.220 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", Ignoring malparsed backtrace: 2022-09-04T14:39:47.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:47.243 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:47.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", Ignoring malparsed backtrace: 2022-09-04T14:39:47.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:47.266 INFO:teuthology.orchestra.run.smithi163.stdout: { , 2022-09-04T14:39:47.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "journaled_completion_queued", Ignoring malparsed backtrace: 2022-09-04T14:39:47.289 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:47.289 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ , 2022-09-04T14:39:47.290 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:47.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7114000000000002e-05 , 2022-09-04T14:39:47.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:47.312 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1732999999999999e-05 , 2022-09-04T14:39:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: }, , 2022-09-04T14:39:47.335 INFO:teuthology.orchestra.run.smithi163.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { , 2022-09-04T14:39:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", , 2022-09-04T14:39:47.358 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ Ignoring malparsed backtrace: 2022-09-04T14:39:47.381 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2022-09-04T14:35:17.271794+0000", , 2022-09-04T14:39:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1479e-05 , 2022-09-04T14:39:47.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:47.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 , 2022-09-04T14:39:47.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:47.831 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:47.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9437000000000001e-05 , 2022-09-04T14:39:47.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:47.859 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:47.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9204000000000001e-05 , 2022-09-04T14:39:47.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:47.887 INFO:teuthology.orchestra.run.smithi038.stdout: { Ignoring malparsed backtrace: 2022-09-04T14:39:47.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.682818+0000", , 2022-09-04T14:39:47.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000195751 , 2022-09-04T14:39:47.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:47.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:29.741702+0000", , 2022-09-04T14:39:47.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7e-05 , 2022-09-04T14:39:47.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, Ignoring malparsed backtrace: 2022-09-04T14:39:47.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", , 2022-09-04T14:39:47.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:39.910400+0000", , 2022-09-04T14:39:47.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 Ignoring malparsed backtrace: 2022-09-04T14:39:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", , 2022-09-04T14:39:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.962852+0000", , 2022-09-04T14:39:47.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9299999999999999e-06 Ignoring malparsed backtrace: 2022-09-04T14:39:48.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", , 2022-09-04T14:39:48.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:45.965867+0000", , 2022-09-04T14:39:48.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9871000000000007e-05 Ignoring malparsed backtrace: 2022-09-04T14:39:48.032 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:48.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", , 2022-09-04T14:39:48.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.057509+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:48.057 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:48.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", , 2022-09-04T14:39:48.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2022-09-04T14:34:51.121616+0000", Ignoring malparsed backtrace: 2022-09-04T14:39:48.081 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:34:51.077956+0000", , 2022-09-04T14:39:48.081 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 296.744325043, , 2022-09-04T14:39:48.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013742661999999999, Ignoring malparsed backtrace: 2022-09-04T14:39:48.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:48.106 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:48.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", Ignoring malparsed backtrace: 2022-09-04T14:39:48.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, , 2022-09-04T14:39:48.131 INFO:teuthology.orchestra.run.smithi038.stdout: { , 2022-09-04T14:39:48.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "write_thread_in_journal_buffer", Ignoring malparsed backtrace: 2022-09-04T14:39:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:932 3.6s0 3:630e74fd:::smithi16339483-16:head [copy-from ver 70] snapc 3b=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", , 2022-09-04T14:39:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:35:21.187265+0000", , 2022-09-04T14:39:48.156 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 266.63501625599997, Ignoring malparsed backtrace: 2022-09-04T14:39:48.180 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4557.0:933 3.6s0 3:630e74fd:::smithi16339483-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", , 2022-09-04T14:39:48.181 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2022-09-04T14:35:21.187311+0000", , 2022-09-04T14:39:48.181 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 266.63497012599998, Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:39:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:39:50.439 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , 2022-09-04T14:39:54.100 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: 2022-09-04T14:39:54.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight , 2022-09-04T14:39:54.607 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:39:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:39:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:39:57.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:39:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:39:58.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:40:01.845 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops , 2022-09-04T14:40:01.925 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: 2022-09-04T14:40:02.433 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:05.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight Ignoring malparsed backtrace: 2022-09-04T14:40:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:06.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops , 2022-09-04T14:40:06.100 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:40:10.277 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , 2022-09-04T14:40:13.938 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: 2022-09-04T14:40:14.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops , 2022-09-04T14:40:14.447 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:40:17.610 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:40:21.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight , 2022-09-04T14:40:21.780 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: 2022-09-04T14:40:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:25.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight , 2022-09-04T14:40:25.949 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:30.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:40:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:33.732 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:40:34.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops , 2022-09-04T14:40:34.290 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: 2022-09-04T14:40:34.801 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:37.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_b85c65e63903c9c9c3e7a140fe41a7c609105ec3/qa/tasks/ceph_manager.py", line 3057, in revive_osd , timeout=timeout, stdout=DEVNULL) , File "/home/teuthworker/src/git.ceph.com_ceph-c_b85c65e63903c9c9c3e7a140fe41a7c609105ec3/qa/tasks/ceph_manager.py", line 1985, in wait_run_admin_socket Ignoring malparsed backtrace: OSError: Socket is closed , 2022-09-04T14:40:39.379 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: 2022-09-04T14:40:39.889 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:42.444 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight Ignoring malparsed backtrace: 2022-09-04T14:40:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:43.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops , 2022-09-04T14:40:43.041 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:40:46.705 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: OSError: Socket is closed , 2022-09-04T14:40:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:49.291 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops Ignoring malparsed backtrace: 2022-09-04T14:40:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:49.870 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight , 2022-09-04T14:40:49.901 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:40:53.563 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: 2022-09-04T14:40:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:40:56.673 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops , 2022-09-04T14:40:56.723 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:40:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:41:00.384 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: 2022-09-04T14:41:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:03.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops , 2022-09-04T14:41:03.540 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:41:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:41:07.200 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: 2022-09-04T14:41:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:10.294 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight , 2022-09-04T14:41:10.357 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:41:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:41:14.022 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:41:17.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight , 2022-09-04T14:41:17.681 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:41:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:41:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:20.815 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:41:24.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2022-09-04T14:41:24.504 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:41:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:27.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: 2022-09-04T14:41:31.235 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops , 2022-09-04T14:41:31.315 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): Ignoring malparsed backtrace: 2022-09-04T14:41:31.824 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:41:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:34.395 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused Ignoring malparsed backtrace: OSError: Socket is closed , admin_socket: exception getting command descriptions: [Errno 111] Connection refused , 2022-09-04T14:41:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 22 Ignoring malparsed backtrace: 2022-09-04T14:41:38.630 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all Ignoring malparsed backtrace: self._write_all(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) Ignoring malparsed backtrace: self.channel.sendall(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) Ignoring malparsed backtrace: return self._send(s, m) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: 2022-09-04T14:41:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:41.754 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops , 2022-09-04T14:41:41.785 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write , self._write_all(data) Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall , sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send , raise socket.error("Socket is closed") , OSError: Socket is closed Ignoring malparsed backtrace: 2022-09-04T14:41:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:45.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops , 2022-09-04T14:41:45.239 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: 2022-09-04T14:41:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-04T14:41:45.737 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight , 2022-09-04T14:41:45.747 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 108, in signal , self.proc.stdin.write(struct.pack('!b', sig)) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 523, in _write_all , count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write Ignoring malparsed backtrace: count = self._write(data) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write , self.channel.sendall(data) Ignoring malparsed backtrace: sent = self.send(s) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send , return self._send(s, m) Ignoring malparsed backtrace: raise socket.error("Socket is closed") , OSError: Socket is closed , 2022-09-04T14:41:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 124 Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/git.ceph.com_ceph-c_b85c65e63903c9c9c3e7a140fe41a7c609105ec3/qa/tasks/rados.py", line 286, in task , running.get() Ignoring malparsed backtrace: yield vars , File "/home/teuthworker/src/git.ceph.com_ceph-c_b85c65e63903c9c9c3e7a140fe41a7c609105ec3/qa/tasks/ceph.py", line 1902, in task , ctx.managers[config['cluster']].stop_pg_num_changes() Ignoring malparsed backtrace: r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() Ignoring malparsed backtrace: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run , r.wait() Ignoring malparsed backtrace: 2022-09-04T14:47:50.555 DEBUG:teuthology.orchestra.run.smithi155:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq , gzip: /var/log/ceph/valgrind/*.gz: No such file or directory , gzip: /var/log/ceph/valgrind/*.gz: No such file or directory Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_b85c65e63903c9c9c3e7a140fe41a7c609105ec3/qa/tasks/rados.py", line 286, in task , running.get() , File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get Ignoring malparsed backtrace: 2022-09-04T14:47:55.634 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2022-09-04T14:47:55.658 INFO:teuthology.task.internal.syslog:Compressing syslogs... , 2022-09-04T14:47:55.660 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- Ignoring malparsed backtrace: proc = self.run(**kwargs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs)